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1 . 0  INTRODUCTION 


1.1  BACKGROUND 

The  Federal  Aviation  Administration  (FAA)  is  involved  in  a  major 
modernization  and  expansion  of  the  National  Airspace  System  (NAS) 
in  order  to  meet  future  requirements  and  demands.  The  Advanced 
Automation  System  (AAS)  is  estimated  to  contain  two  million  lines 
of  code.  The  primary  AAS  language  is  Ada.  While  the  AAS  is  the 
largest  of  the  NAS  subsystems,  many  other  subsystems  such  as  Data 
Link,  Mode-S,  NADIN  II,  CWP,  and  AERA  are  software  intensive  as 
well.  More  and  more  NAS  functions  will  be  automated  and  software 
will  be  increasingly  depended  upon. 

Key  to  the  NAS  modernization  initiative  is  the  acquisition  of 
software  to  support  the  intended  services.  Considering  the 
complexity  of  the  effort,  it  is  imperative  that  the  FAA  methodology 
used  to  acquire  and  maintain  NAS  software  be  documented  to  provide 
a  common  perspective  for  planning,  requirements  analysis,  system 
design,  program  development  and  implementation,  test  ana 
evaluation,  deployment,  and  maintenance. 


1.2  SCOPE,  OBJECTIVES  AND  GOALS 

The  System  Engineering  and  Program  Management  Office,  System 
Design  and  Configuration  Management  Division  (ASE-200) ,  is 
responsible  for  developing  and  maintaining  the  technical  standards 
used  in  acquiring  NAS  subsystems.  To  date,  FAA  has  standardized  on 
a  software  development  standard,  FAA-STD-026  (based  on  DoD- 
STD-2167A),  and  on  the  Ada  Programming  language.  In  implementing 
these  standards  several  issues  have  arisen  pertaining  to  the 
differences  between  software  development,  testing,  and  maintenance 
practices  between  the  NAS  acquisition  and  maintenance 
organizations . 

To  clarify  and  address  these  issues  the  FAA  tasked  Technology 
Planning  Incorporated  (TPI)  to  review  the  existing  FAA  software 
policies,  orders,  standards,  process,  and  procedures  which  are  used 
by  the  FAA  for  NAS  software  acquisition,  maintenance,  data  and 
documentation  management.  The  objectives  were  to: 

1.  Identify  current  deficiencies,  omissions,  and  conflicts  with 
respect  to  these  guidelines; 

2.  Review  specific  NAS  Plan  subsystems  in  terms  of  the  proposed 
end-state  software,  data  engineering  environment,  software 
development  methodology,  general  implementation  and 
maintenance  strategy; 
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3.  Document  deficiencies  and  make  recommendations;  and 

4.  Assist  in  the  preparation  of  a  NAS  System-level  Plan  to 
address  the  identified  deficiencies. 

As  a  result  of  accomplishing  the  above  objectives,  the  following 
longer  range  goals  can  be  initiated. 

Goal  1 


Improve  the  process  of  constructing  quality 
software . 


Goal  2 


Reduce  the  risk  factors  associated  with  building 
systems.  The  risk  factors  include  technical, 
schedule,  cost,  operational  and  support  areas. 

Goal  3 


Heighten  the  awareness  and  increase  the  involvement 
of  management  and  other  appropriate  staff  with  the 
software  acquisition  process. 

Goal  4 

Cultivate  the  development  of  software  engineering 
strength  within  FAA . 


1.3  TERMINOLOGY 

Throughout  this  report  the  word  "guidelines"  is  used  to  indicate 
any  of  the  standards,  orders,  policies  or  procedures  used  by  the 
FAA  for  NAS  software  acquisition. 


1.4  METHODOLOGY 

In  order  to  adequately  address  Software  Acquisition  by  the  FAA,  the 
acquisition  of  software  must  be  looked  at  in  the  larger  context  of 
the  NAS  System  Life  Cycle  and  must  consider  the  system  level 
aspects  of  the  life  cycle  which  directly  influence  or  are 
influenced  by  software. 

The  methodology  used  in  performance  of  this  task  includes  four 
steps : 

1 .  Development  of  a  Data  Collection  Instrument  which  was  used 
during  interviews  with  FAA  personnel  who  are  responsible  for 
some  aspect  of  software  within  some  phase  of  the  NAS  System 
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Life  Cycle; 

2.  Data  collection  which  included  the  personnel  interviews  and 
reviews  of  selected  policies,  orders,  standards  and 
guidelines; 

3.  Analysis  of  the  data  collected; 

4.  Development  of  conclusions  and  recommendations  which  have 
resulted  in  a  plan  of  action  for  the  FAA. 

The  following  organizations  participated  in  the  interviews  for  this 
task : 

AAF-4,  AAP-120,  AAP-220,  AAP-320,  AAP-400,  AAT-14,  ACD-340, 
ACD-350,  ACM -110,  ACN-130,  ACN-210,  ACN-310,  ACS-320,  ADS-120, 
AHT-400,  AHT-500,  ALG-410,  AMC-300,  AOR-110,  APS-300,  APS-500, 
ASA-6,  ASA-130,  ASA-210,  ASM-140,  ASM-160,  ATR-210,  ATR-250, 
LOGICON. 


2.0  OVERALL  ASSESSMENT 

One  of  the  most  common  problems  facing  government  and  private 
organizations  is  the  transition  to  a  modern  and  effective  software 
engineering  methodology  for  creation  of  software  systems, 
acquisition  of  software  systems,  or  both.  This  problem  is 
addressed  in  depth  in  the  literature  ( [AFSB89] ,  [AFSC89] ,  [NASA89] , 
and  [PRESS88 ] )  but  still  remains  a  problem.  In  these  and  other 
studies  the  fundamental  approach  has  been  a  four  stage  approach 
consisting  of: 

1.  Assessment  of  current  practices; 

2.  Development  of  a  strategic  plan  for  the  transition  to  improved 
practices; 

3.  Implementation  of  the  plan; 

4.  Evaluation  of  the  success  or  failure  of  the  plan. 

TPI  has  performed  the  first  portion  of  step  1  by  performing  a 
qualitative  analysis  of  the  FAA' s  software  acquisition  process.  The 
findings  of  this  analysis  can  serve  as  the  basis  for  identifying 
the  remaining  activities  required  to  improve  the  FAA' s  software 
acquisition  process. 

The  results  of  TPI's  assessment  are  summarized  as  follows: 

1 .  There  is  a  lack  of  adequate  commitment  to  the  software  aspects 
of  FAA  projects; 
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2.  There  is  not  enough  up  to  date  software  expertise  within  the 
FAA; 

3.  Some  problems  in  software  acquisition  are  due  to  the  FAA 
organizational  structure  which  artificially  separates 
hardware,  software  and  systems  engineering  activities; 

4.  There  are  problems  that  are  the  result  of  inadequate,  missing, 
or  poorly  understood  FAA  standards,  guidelines  and  orders 
associated  with  the  software  acquisition  process. 

5.  There  is  a  lack  of  involvement  of  software  maintenance  people 
during  the  program  definition  and  software  acquisition  phases. 


Although  the  report  that  TPI  produced,  detailing  the  findings  of 
the  assessment,  reports  several  other  problems,  analysis  shows  that 
almost  all  the  problems  can  be  traced  back  to  these  four  basic 
points . 


3.0  RECOMMENDED  COURSE  OF  ACTION 

To  continue  with  the  approach  outlined  in  Section  2.0  above,  TPI 
recommends  the  toll owing  course  of  action.  Although  the  activities 
detailed  imply  a  sequential  order,  there  is  room  for  parallel 
activities  to  occur.  Some  of  these  activities  are  required  to  be 
performed  in  the  near  term,  while  others  may  be  performed  in  the 
range  from  medium  to  long  term. 

While  a  complete  implementation  of  all  the  recommendations  is 
encouraged,  budget  considerations  may  dictate  that  selected  parts 
of  the  recommendations  be  deferred  to  a  later  time.  This  is 
entirely  possible  due  to  the  nature  of  the  problems.  There  will 
be  a  loss  of  the  synergistic  effect  of  implementing  all 
recommendations  as  one  program  plan.  Due  to  the  complexity  of  the 
software  acquisition  process,  it  is  expected  that  a  complete 
program  to  improve  the  process  will  take  several  years  to 
accomplish . 

The  following  activities  have  been  identified  to  address  the 
approach  and  the  problems  cited  in  the  preceding  section.  Each 
activity  supports  the  longer  range  goals  described  in  Section  1.2. 
The  predominant  goal  for  the  activity  is  listed  along  with  a  brief 
explanation . 


3.1  ESTABLISHING  AN  ADEQUATE  SOFTWARE  COMMITMENT 

The  FAA  needs  to  establish  a  clear  and  firm  commitment  to  dealing 
with  software  issues  in  FAA  projects.  Such  a  commitment  consists 
of:  (1)  allocation  of  resources  and  funds  to  improve  the  process 
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and  staff  capabilities.*  (2)  establishment  of  additional  and 
enhanced  software  policies;  (3)  participation  with  other 
organizations  dealing  with  the  same  issues;  and  (4)  encouraging  the 
involvement  of  senior  management.  This  activity  supports  Goal  3  bv 
elevating  the  level  and  degree  of  attention  given  to  software 
projects . 


3.1.1  Software  Policy  Statement 

The  FAA  needs  to  develop  a  comprehensive  software  policy  statement 
outlining  the  goals,  commitment,  and  rules  pertaining  to  software 
systems  acquisition.  This  software  policy  statement  will  be  used 
to  perform  evaluations  of  FAA  projects  and  also  to  evaluate 
technologies,  tools,  and  FAA  guidelines  pertaining  to  software 
projects.  Such  a  policy  will  be  coupled  with  improved  guidelines, 
standards,  and  orders. 

Because  of  poor  experiences  with  past  FAA  projects,  most  projects 
that  have  a  significant  software  component  should  be  considered 
high  risk  projects.  The  overall  policy  statement  will  require  a 
software  risk  assessment  and  risk  management  plan  as  an  integral 
part  of  these  projects.  This  activity  supports  Goal  2  bv  providing 
a  method  for  evaluating  high  risk  projects  during  the  early  phases 
of  the  software  acquisition  life  cycle. 


3.1.2  Establishment  of  Technical  Support  Groups 

The  FAA  needs  to  establish  agency-wide  support  groups  dealing  with 
major  software  issues.  Such  groups  will  both  support  projects  and 
also  serve  as  review  and  evaluation  groups  for  project  activities. 
They  will  be  available  upon  request  to  assist  a  project  with 
project  definition  and  initiation  activities.  These  groups  will 
serve  as  the  focal  points  for  infusing  software  technology  ii._o  the 
FAA.  They  will  interact  with  industry  (especially  contractors) , 
academia,  and  other  government  agencies  dealing  with  the  same 
issues  (e.g.,  DoD,  NASA).  These  groups  will  support  the  infusion 
of  software  skills  into  the  FAA  strff  and  projects  through  such 
activities  as:  training,  workshops,  symposia,  and  forums. 

It  is  anticipated  that  these  groups  will  be  small  in  size,  between 
3  and  7  persons,  and  that  these  groups  will  report  to  a  high  enough 
management  level  to  firmly  establish  th»  validity  of  their 
missions . 


3. 1.2.1  Software  Process  Support  Working  Groups  -  These  groups 
will  deal  with  the  overall  software  process  including  guidelines, 
standards,  and  orders  as  well  as  with  software  project  management. 
These  groups  deal  primarily  with  software  project  management  rather 
than  specific  technologies.  Software  quality  assurance  and 


Page  5 


software  productivity  also  fall  under  these  groups.  These  groups 
are  intended  to  be  temporal,  in  nature  and  to  focus  on  one  software 
task  at  a  time  in  order  to  resolve  immediate  issues  or  concerns. 
This  activity  supports  Goal  l  by  providing  a  feedback  loop  during 
the  software  construction  process. 


3. 1.2. 2  Software  Technology  Support  Group  -  This  group  will  deal 
with  specific  software  technologies  and  tools,  as  well  as  provide 
overall  guidance  to  the  working  groups  discussed  above.  This 
group's  charter  is  to  identify  those  technologies,  both  available 
and  emerging,  that  are  applicable  to  the  FAA' s  environment.  The 
services  of  this  group  would  be  provided  on  a  request  basis  and 
would  be  available  as  an  on-going  support  function.  Further,  this 
group  will  be  responsible  for  the  eventual  incorporation  of  the 
appropriate  technologies  into  the  FAA  environment.  This  activity 
supports  Goal  1  by  providing  the  technologies  and  tools  necessary 
for  the  development  and  maintenance  process. 


3 . 2  IN-DEPTH  TECHNICAL  ASSESSMENT  OF  FAA  SOFTWARE  CAPABILITIES 

Although  a  qualitative  assessment  of  the  FAA' s  software 
acquisition  process  and  maintenance  has  already  been  performed  by 
TP  I ,  an  in-depth  quantitative  assessment  needs  to  be  performed  to 
guide  any  corrective  action.  For  example,  an  assessment  similar  to 
those  presented  in  the  literature  ([HUMP87]  and  [PRESS88 ] ) ,  which 
addresses  the  software  engineering  environment  and  software 
management  activities,  needs  to  be  tailored  for  the  FAA.  Such  an 
assessment  will  evaluate  both:  (1)  the  technology  present  within 
the  FAA  and  also  (2)  the  software  process  itself.  This  study  will 
involve  a  comprehensive  survey/skills  profile  inventory  of 
software  engineering  skills  present  within  FAA  personnel.  TPI's 
initial  study  identified  a  lack  of  up  to  date  software  skills 
within  the  FAA  as  a  serious  problem.  This  follow-on  assessment 
will  quantify  the  skills  currently  present  within  the  FAA  and 
identify  precisely  those  areas  where  skills  are  lacking. 


3.2.1  Software  Process  Assessment 

A  software  process  assessment  evaluates  the  current  practices  with 
respect  to  those  tools,  methods,  standards,  and  processes  used  by 
the  software  engineers  in  performance  of  these  assignments. 

Performing  a  software  process  assessment  will  evaluate  the  FAA' s 
software  process  for  acquisitions  and  maintenance  against  the 
recommended  practices  established  within  the  software  engineering 
community.  Areas  lacking  adequate  emphasis  along  with  the  skills 
necessary  to  pro-vide  that  emphasis  will  be  identified.  Although 
the  FAA' s  software  acquisition  and  maintenance  needs  are  similar  to 
other  organizations,  some  deviations  from  the  normal  practices  are 
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expected  and  acceptable.  The  framework  for  evaluation  of  the  FAA' s 
process  should  be  extended  to  also  evaluate  the  various  contr:  ctors 
supporting  the  FAA' s  projects.  This  activity  supports  Goal  1  by 
establishing  and  refocusing  the  software  process  for  acquisitions 
and  Goal  2  by  reducing  the  exposure  for  FAA  by  identifying  risks 
early  in  contractor's  projects. 


3.2.2  Software  Technology  Assessment 

Unlike  the  previous  software  process  assessment  that  deals  with 
steps  and  phases  of  a  software  project  life  cycle,  this  software 
technology  assessment  deals  with  evaluating  the  details  of  each 
step  in  the  life  cycle.  For  example,  the  previous  assessment 
concerns  itself  with  the  existence  of  a  coding  standard  while  this 
step  will  evaluate  the  quality  of  such  a  standard. 

This  step  will  also  perform  an  inventory  of  the  software  skills  of 
the  FAA  staff  and  compare  them  against  the  skills  identified  by  the 
process  assessment  as  necessary  for  the  various  software 
engineering  roles  (i.e.  manager, analyst ,  designer,  coder,  tester). 
This  activity  supports  Goal  4  by  providing  a  skill  inventory  for 
software  engineers.  The  skill  inventory  would  provide  the  basis 
for  a  professional  development  plan  for  each  person. 


3.3  OVERCOMING  ORGANIZATIONAL  BARRIERS 

The  management  within  the  FAA  needs  to  recognize  and  acknowledge 
that  the  NAS  life  cycle  software  process  has  difficulties  which 
are  associated  with  the  current  organizational  structure.  The 
hardware,  software,  and  systems  engineering  activities  are,  in  most 
cases,  separated  due  to  organizational  decisions.  This  separation 
causes  daily  coordination  problems  for  those  personnel  involved  in 
these  engineering  activities.  Lack  of  appropriate  coordination 
leads  to  incorrect  solutions  with  respect  to  requirements 
definition,  project  implementation,  system  integration  and  post¬ 
deployment  support. 


Policies  and  procedures  which  overcome  the  organizational  barriers 
need  to  be  implemented  and  rigorously  enforced  by  all  levels  of  FAA 
management.  Such  policies  will  consider:  (1)  increasing  awareness 
of  FAA  personnel  with  respect  to  the  entire  NAS  life  cycle's 
products,  reviews,  and  activities;  (2)  containment  of  life  cycle 
costs  as  the  responsibility  of  all  organizations  who  have  any  level 
of  involvement  in  the  NAS  system  life  cycle;  (3)  smooth  transition 
procedures;  (4)  <=>arly  consideration  of  the  needs  of  all 
organizations  (during  project  definition);  and  (5)  high  level 
commitment  to  enforcement  of  these  policies.  This  activity 
supports  Goal  2  by  providing  management  with  estimated  cost  and 
schedule  risks  for  each  project. 
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3.4  DEVELOPMENT  AND  IMPLEMENTATION  OF  A  STRATEGIC 

PLAN  FOR  IMPROVING  THE  NAS  SYSTEM  LIFE  CYCLE 

This  plan  will  identify  the  steps  required  to  improve  the 
acquisition  of  software  systems  within  the  FAA.  It  will  identify 
activities  such  as:  (1)  improvements  to  FAA  standards,  orders,  and 
guidelines;  (2)  establishment  of  a  software  project  and  system 
engineering  curriculum;  (3)  improvements  to  the  requirements 
determination  phase  of  the  life  cycle;  and  (4)  improvements  to  the 
the  phase  transitions  during  the  life  cycle.  The  Strategic  Plan 
for  Improving  The  NAS  System  Life  Cycle  will  consist  of  three  major 
activities : 

1.  Software  technology  transfer  into  the  FAA; 

2.  Improvements  to  the  software  process  through  revisions  of 
FAA  standards,  guidelines,  and  orders; 

3.  Continued  evaluation  of  the  software  acquisitions  process 
and  software  productivity. 

When  the  plan  is  completed,  a  cost-benefit  analysis  will  be 
performed  to  determine  the  implementation  priority  for  the  various 
aspects  of  the  plan.  This  activity  supports  Goal  1  by  focusing, 
monitoring  and  re-tuning  the  software  acquisition  process. 


3.4.1  Software  Technology  Transfer 

The  transfer  of  software  technology  into  the  FAA  involves  two 
areas:  training  and  support  systems. 


3. 4. 1.1  Training  -  The  strategic  plan  will  first  develop  a 
software  engineering  education  curriculum  that  covers  all  aspects 
of  software  education.  Such  education  will  consist  of  inhouse 
courses  and  seminars,  off-site  commercial  offerings,  educational 
support  for  staff  members,  and  sponsoring  of  relevant  workshops  and 
symposia.  The  separate  training  needs  of  management,  technical 
staff,  and  project  specific  requirements  will  be  identified  and 
accommodated  by  the  proposed  curriculum.  The  various  roles 
(analyst,  designer,  coder,  tester)  identified  in  the  software 
process  assessment  will  serve  as  one  of  the  inputs  to  this 
process.  This  activity  supports  Goal  4  by  assisting  with  the 
development  of  a  curriculum  to  support  the  technical  growth  of 
staff  . 


3. 4. 1.2  Tool  and  Support  Systems  -  Available  technological 
advances  include  software  tools  as  well  as  techniques.  The 
strategic  plan  will  identify  a  means  of  evaluating  software  tools 
and  environments  that  would  prove  beneficial  to  the  FAA  and  then 
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provide  a  plan  for  making  these  tools  and  environments  available 
within  the  FAA.  In  some  cases,  evaluation  experiments,  perhaps 
involving  real  projects,  will  be  performed  to  assess  the  value  of 
these  tools.  This  activity  supports  Goal  4  bv  evaluating  the  tools 
needed  for  staff  involved  with  the  software  acquisition  process. 


3.4.2  Improving  the  Software  Process  Through  Standards. 

Orders,  and  Guidelines 

While  many  of  the  FAA' s  standards,  orders,  and  guidelines  are 
currently  adequate,  several  problems  were  evident  during  TPI's 
initial  assessment.  The  strategic  plan  will  outline  how  the 
missing  and  inadequate  standards  are  to  be  improved.  The  software 
process  assessment  may  identify  additional  missing  standards, 
orders,  and  guidelines.  The  plan  will  also  address  how  appropriate 
information  concerning  FAA  standards,  orders  and  guidelines  is  to 
be  disseminated  to  the  FAA  staff.  The  plan  will  also  address  by 
what  process  the  standards,  orders,  and  guidelines  will  be  kept 
current.  This  activity  supports  Goal  1  by  supporting  the 
improvement,  specification  and  use  of  standards,  orders  and 
guidelines  and  Goal  3  by  elevating  the  software  process  to  the 
attention  of  management. 


3. 4. 2.1  Improvement  to  Standards,  Orders,  and  Guidelines  -  Several 
FAA  and  other  standards,  orders,  and  guidelines  were  identified  as 
causing  difficulties  with  NAS  software  acquisition.  FAA-STD-026 
and  DoD-STD-2167A  were  foremost  in  this  area.  The  plan  will 
identify  the  strategy  for  updating  or  replacing  some  standards  with 
more  effective  standards.  DoD-STD-2167A  is  now  an  accepted 
standard  throughout  the  government.  However,  no  consistent  set  of 
guidelines  for  tailoring  2167A  and  for  using  2167A  for  maintenance 
operations  are  available.  The  plan  will  include  an  activity  to 
provide  support  and  guidance  for  the  use  of  DoD-STD-2167A  within 
the  FAA  framework.  The  plan  will  also  direct  the  development  of  a 
strategy  for  the  retrofitting  of  standards,  for  purposes  of 
maintenance,  where  appropriate.  Others  in  the  software 
engineering  field  have  reported  success  with  retrofitting  a 
tailored  version  of  DoD-STD-21 67A  for  purposes  of  software 
maintenance  [KEMP88]  to  projects  not  originally  using  2167A.  The 
strategic  plan  will  address  such  issues  since  maintenance  was 
identified  as  a  problem  area  within  the  FAA. 


3. 4. 2. 2  Specification  of  FAA  Standards.  Orders,  and  Guidelines  - 
The  plan  will  address  the  improvement  and  dissemination  of 
guidelines  for  using  the  standards,  orders,  and  guidelines  within 
the  FAA  projects.  It  should  be  clear  to  the  staff  which  standards, 
orders,  and  guidelines  must  be  applied  to  their  project  and  these 
rules  should  be  enforced.  Of  course  waivers  are  possible,  but  the 
rules  should  be  consistently  applied  across  FAA  projects.  One  of 
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the  recurring  themes  found  in  the  use  of  standards  outside  the  FAA 
is  the  use  of  tailoring  and  waivers.  Properly  used,  this  can 
prevent  the  misapplication  of  standards. 


3. 4. 2. 3  Improvement  of  the  Use  of  FAA  Standards.  Orders,  and 
Guidelines  -  After  the  FAA' s  standards,  orders,  and  guidelines 
have  been  strengthened,  there  must  be  some  mechanism  for 
distributing  this  information  to  the  FAA  staff.  The  strategic  plan 
will  identify  methods  of  disseminating  information  about  FAA 
standards,  orders,  and  guidelines  to  appropriate  FAA  staff 
members . 

Possible  methods  for  providing  such  distribution  might  include: 
training  courses;  creation  of  an  on-line  database  system  relating 
the  stages  of  the  NAS  life  cycle  to  appropriate  FAA  standards, 
orders,  and  guidelines;  development  of  quick  reference  charts;  and 
development  of  a  software  manager's  handbook  providing  further 
explanation  concerning  the  use  of  FAA  standards,  orders,  and 
guidelines  in  relation  to  the  NAS  life  cycle. 

Another  way  to  improve  the  effective  use  of  the  standards  is  to 
ensure  that  the  procurement  guidelines  for  software  acquisition 
incorporate  these  new  standards. 


3.4.3  Evaluation 

Finally,  the  strategic  plan  will  direct  the  evaluation  of  current 
and  existing  software  practices  within  the  FAA.  This  involves  the 
establishment  and  specification  of  software  program  management  and 
engineering  metrics  based  on  the  FAA  Software  Policy  Statement 
mentioned  in  Section  3.1.1  of  this  summary.  That  will  provide  a  way 
of  measuring  whether  or  not  the  expectations  are  being  met.  As  the 
plan  is  implemented,  the  degree  of  success  or  failure  can  be 
determined  by  re-evaluating  the  FAA' s  software  activities  and 
analyzing  the  actual  results  against  the  expected  results. 
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1.0  INTRODUCTION 


1.1  BACKGROUND 

The  Federal  Aviation  Administration  (FAA)  is  involved  in  a  major 
modernization  and  expansion  of  the  National  Airspace  System  (NAS) 
in  order  to  meet  future  requirements  and  demands.  The  Advanced 
Automation  System  (AAS)  is  estimated  to  contain  two  million  lines 
of  code.  The  primary  AAS  language  is  Ada.  While  the  AAS  is  the 
largest  of  the  NAS  subsystems,  many  other  projects  such  as  Data 
Link-s,  NADIN  II,  CWP  and  AERA  III  are  software  intensive,  and  more 
and  more  NAS  functions  will  be  automated  via  software. 

Key  to  the  NAS  modernization  initiative  is  the  acquisition  of 
software  to  support  the  intended  services.  Considering  the 
complexity  of  the  effort,  it  is  imperative  that  the  FAA  methodology 
used  to  acquire  and  maintain  NAS  software  be  documented  to  provide 
a  common  perspective  for  planning,  requirements  analysis,  system 
design,  program  development  and  implementation,  test  and 
evaluation,  deployment,  and  maintenance. 


1.2  SCOPE.  OBJECTIVES  AND  GOALS 

The  System  Engineering  and  Program  Management  office.  System  Design 
and  Configuration  Management  Division  (ASE-200) ,  is  responsible  for 
developing  and  maintaining  the  technical  standards  used  in 
acquiring  NAS  subsystems.  To  date,  FAA  has  standardized  on  a 
software  development  standard,  FAA-STD-026  (based  on  DoD-STD- 
2167A),  and  on  the  Ada  programming  language.  In  implementing  these 
standards  several  issues  have  arisen  pertaining  to  the  differences 
between  software  development,  and  maintenance  practices  between  the 
NAS  acquisition  and  maintenance  organizations. 

To  clarify  and  address  these  issues,  the  FAA  has  tasked  Technology 
Planning  Incorporated  (TPI)  to  review  the  existing  FAA  software 
engineering  process,  standards,  policies,  procedures  and  orders 
which  are  used  by  the  FAA  for  NAS  software  acquisition, 
maintenance,  data  and  documentation  management.  The  objectives  are 
to:  (1)  identify  current  deficiencies,  omissions,  and  conflicts 
with  respect  to  these  standards,  policies,  procedures  and  orders; 
(2)  review  specific  NAS  Plan  subsystems  in  terms  of  the  proposed 
end-state  software,  data  engineering  environment,  software 
development  methodology,  general  implementation  and  maintenance 
strategy;  (3)  document  deficiencies  and  make  recommendations;  and 
(4)  assist  in  the  preparation  of  a  NAS  System-level  Plan  to  address 
the  identified  deficiencies. 

As  a  result  of  accomplishing  the  above  objectives,  the  following 
long  range  goals  can  be  initiated. 
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Goal  1 


Improve  the  process  of  constructing  quality 
software . 


Goal  2 


Reduce  the  risk  factors  associated  with  building 
systems.  The  risk  factors  include  technical, 
schedule,  cost,  operational  and  support  areas. 

Goal  3 


Heighten  the  awareness  and  increase  the  involvement 
of  management  and  other  appropriate  staff  with  the 
software  acquisition  process. 

Goal  4 


Cultivate  the  development  of  software  engineering 
strength  within  FAA. 


1.3  TERMINOLOGY 

Throughout  this  report  the  use  of  the  word  "guide lines"  is  used  to 
indicate  any  of  the  standards,  orders,  policies  or  procedures  used 
by  the  FAA  for  NAS  software  acquisition. 
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2.0  METHODOLOGY 


In  order  to  adequately  address  Software  Acquisition  by  the  FAA,  the 
acquisition  of  software  must  be  looked  at  in  the  larger  context  of 
the  NAS  System  Life  Cycle  and  must  consider  the  system  level 
aspects  of  the  life  cycle  which  directly  influence  or  are 
influenced  by  software. 

The  methodology  used  during  Task  Order  0011  is  described  in  the 
following  paragraphs  and  includes  four  steps: 


(1) 

development 

of  a  Data  Collection 

Instrument, 

(2) 

data  collection. 

(3) 

analysis  of 

the  data  collected, 

(4) 

development 
result  in  a 

of  conclusions  and  recommendations  which 
plan  of  action  for  the  FAA. 

2.1  STEP  1  -  DATA  COLLECTION  INSTRUMENT 

The  Data  Collection  Instrument  contains  a  list  of  questions  to  be 
used  during  interviews,  a  NAS  System  Life  Cycle  definition,  and  a 
list  of  the  current  FAA  standards,  FAA  orders,  MIL  standards,  and 
DoD  standards  believed  to  be  in  use  by  the  FAA  for  software 
acquisition.  This  instrument  will  be  updated  and  modified  as 
required  during  the  interview  process.  The  life  cycle  definition 
has  been  updated  since  delivery  of  the  Data  Collection  Instrument 
and  is  shown  in  Figure  1-1  and  expanded  in  Appendix  E. 

Also  as  part  of  step  one,  a  memo  was  written  and  sent  to  all 
concerned  service  level  managers  within  the  FAA,  requesting  names 
of  persons  to  be  interviewed  during  step  two. 


2.1.1  NAS  System  Life  Cycle 

A  brief  description  of  the  various  phases  of  the  NAS  System  Life 
Cycle  as  defined  for  this  report  is  as  follows. 

(1)  Requirements  Determination  -  This  consists  of  two 
distinct  phases,  Concept  Definition  and  Validation  and 
Program  Definition.  The  Concept  Definition  and 
Validation  phase  encompasses  all  of  the  FAA  R,  E&D 
activities  and  results  in  initial  product  specifications 
which  are  input  to  the  Program  Definition  phase.  During 
Program  Definition,  the  Statement  of  Work,  the 
System/Pro ject/Proyi am  Specifications  and  the  Request  for 
Proposal  are  finalized. 
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(2) 


Acquisition  -  This  consists  of  eight  sub-phases.  The 
Project  Initiation  phase  incorporates  the  initial 
activities  of  a  program.  These  include  contract 
negotiations,  finalization  of  the  requirements 
definitions,  and  production  of  initial  management  and 
planning  documentation  by  the  contractor.  Activities 
during  this  phase  must  set  the  stage  for  the  follow-on 
phases  with  respect  to  how  the  FAA  and  contractor  will 
manage  and  control  the  program. The  other  seven  phases  of 
acquisition  are  the  standard  analysis,  preliminary  and 
detailed  design,  development,  and  the  three  testing 
phases . 

(3)  Operational  Support  -  This  phase  covers  the  transition  of 
the  product  from  development  into  operational  use  and 
includes  two  sub-phases,  Operational  Transition  and  Post 
Deployment  Support.  The  Operational  Transition  phase 
includes  those  testing  activities  conducted  by  both  the 
FAA  end  users  and  the  contractor  which  assure  the 
readiness  of  the  product  for  deployment.  The  Post 
Deployment  Support  includes  the  on-going  maintenance  and 
enhancement  activities  which  assure  continuing  operation 
of  a  product  over  many  years  of  service. 


2.2  STEP  2  -  DATA  COLLECTION 

Data  collection  involved  reviewing  the  documentation  and  conducting 
interviews  with  FAA  personnel. 


2.2.1  Documentation  Review 

The  documentation  reviewed  and  reported  on  is  listed  in  Appendix  A 
of  this  report.  Other  references  used  in  performance  of  this  task 
are  listed  in  Appendix  F. 

In  developing  the  list  of  guidelines  for  review,  a  broad  list  of 
FAA  guidelines  were  initially  reviewed  to  determine  whether  they 
were  used  for  NAS  software  acquisition  and  software  management.  The 
number  of  guidelines  was  reduced  to  those  listed  in  Appendix  A. 
Because  of  time  constraints  associated  with  performing  this  task, 
the  selected  guidelines  were  categorized  as  to  whether  they  were 
considered  major  or  not  major  in  terms  of  their  importance  to  NAS 
software  acquisition  and  software  management.  Next  the  guidelines 
were  reviewed  according  to  a  set  of  criteria,  as  discussed  below, 
with  more  emphasis  given  to  the  major  guidelines. 

To  properly  perform  the  guideline  evaluation  process,  a  set  of 
evaluation  criteria  were  established.  These  criteria  were 
prioritized  to  reflect  their  importance  to  the  review  process.  The 
following  evaluation  criteria  were  used  in  the  review  of  the  FAA 
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standards  and  orders.  They  are  listed  in  order  of  decreasing 
priority . 


Evaluation 

Criteria 

Completeness 

Correctness 

Consistency 


Clarity 


Traceability 


Effectiveness 


Flexibility 


Currency 


Comment 

Are  all  required  aspects  of  the 
software  development  process 
covered? 

Are  the  aspects  of  the  software 
development  process  in 
conformance  to  guidelines? 

Are  the  aspects  of  the  software 
development  process  covered  in 
the  same  way  within  each 
document  and  throughout  the 
various  documents? 

Are  the  guidelines  and 
instructions  concise, 
understandable,  and  self- 
contained? 

Does  the  guideline  support  the 
traceability  of  the  quality, 
function,  and  characteristic  of 
an  item  throughout  the  life 
cycle? 

Is  the  guideline  feasible  for 
supporting  implementation  on 
FAA  projects?  Will  the  process 
produce  the  products  needed  to 
manage  the  acquisition  and 
assure  the  quality  and 
correctness  of  the  software? 

Is  the  guideline  adaptable  to 
the  variety  of  projects  and 
approaches  that  will  be 
encountered  by  the  FAA? 

Does  the  guideline  reflect 
modern  software  engineering 
practice''? 


The  review  was  performed  using  a  multi-pass  review  process.  During 
pass  1,  the  effort  concentrated  on  reviewing  each  guideline  on  an 
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individual  (i.e.,  stand-alone)  basis.  During  pass  2,  the  documents 
were  reviewed  to  evaluate  the  consistency  and  completeness  between 
the  various  guidelines . 

The  first  output  of  the  reviews  is  the  summary  report  in  Paragraph 
3.1.1  of  this  report.  The  summary  provides  a  general  evaluation  of 
the  guidelines  and  an  overview  of  the  consistency  and  completeness 
between  the  guidelines.  The  second  output  of  the  reviews  is 
Appendix  B  which  provides  more  detailed  information  concerning  the 
individual  guidelines  that  were  reviewed. 


2.2.2  Personnel  Interviews 

The  FAA  personnel  being  interviewed  are  from  a  variety  of 
organizations  which  have  responsibilities  within  the  phases  of  the 
NAS  System  Life  Cycle.  The  organizations  included  AAP,  ASA,  ADS, 
APS,  ASM,  ATR,  ALG,  AHT,  ACD,  AOR,  ACN,  AAT,  AMC,  ACS,  AAF,  and 
LOGICON. 

The  Data  Collection  Instrument  from  Step  1  was  used  during  the 
interviews  to  guide  the  process  and  to  enable  organization  of  the 
answers.  As  the  interviews  progressed,  the  instrument  was  modified 
as  some  questions  were  found  to  be  inadequate  or  required  more 
detailed  information  than  the  interviewees  were  prepared  to  answer 
given  the  nature  of  the  interview  process. 

The  results  from  the  interviews  were  then  summarized  and  the 
details  were  collected  in  Appendix  C. 


2.3  STEP  3  -  DATA  ANALYSIS 

The  data  analysis  involves  coordination  of  the  information 
collected  during  the  document  reviews  and  the  interviews.  Some  of 
this  information  will  be  organized  using  the  NAS  system  life  cycle 
as  the  guiding  tool.  That  is,  the  information  will  be  categorized 
as  to  the  life  cycle  phase,  product,  action,  or  review  that  it 
affects.  This  categorization  will  then  lead  to  the  last  step  of 
drawing  conclusions  and  making  recommendations. 

Appendix  D  is  a  table  which  lists  the  life  cycle  products,  actions, 
reviews  and  other  relevant  software  engineering  topics  and  shows 
the  guideline  which  is  the  governing  document  with  respect  to  that 
topic.  There  may  be  more  than  one  governing  document  as  the 
Appendix  shows.  The  Appendix  does  not  attempt  to  list  every 
document  which  references  that  topic.  Creation  of  such  a  database 
is  a  recommended  action  for  the  FAA. 

Appendix  E  is  the  Expanded  NAS  System  Life  Cycle  which  itemizes  all 
products,  reviews  and  actions  which  may  apply  to  each  of  the  phases 
and  sub-phases  of  the  life  cycle.  The  items  within  this  Appendix 
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are  the  result  of  the  document  reviews  by  TPI.  This  expanded  life 
cycle  represents  the  picture  found  within  the  current  FAA  NAS 
policy  and  process.  That  is,  all  of  the  documents#  reviews,  and 
actions  itemized  were  found  listed  in  some  FAA  Order  or  standard. 
This  life  cycle  muse  be  tailored  for  each  NAS  project. 


2.4  STEP  4  -  CONCLUSIONS  AND  RECOMMENDATIONS 

The  conclusions  were  then  drawn  and  itemized.  The  final 
recommendations  will  be  in  the  form  of  an  action  plan  for  the  FAA. 
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1 . 0  REQUIREMENTS  DETERMINATION 

1.1  CONCEPT  DEFINITION  AND  VERIFICATION  (R, E&D) 

1.2  PROGRAM  DEFINITION  (MSA:  REQUIREMENTS  DEFINITION) 

2 . 0  ACQUISITION 

2.1  PROJECT  INITIATION  (MSA:  CONCEPT  ANALYSIS) 

2.2  REQUIREMENTS  DEFINITION 

2.3  PRELIMINARY  DESIGN 

2.4  DETAILED  DESIGN  (MSA:  DEMO  PHASE) 

2.5  IMPLEMENTATION  (MSA:  DEMO  PHASE) 

INTEGRATION  AND  TEST  (MSA:  DEMO  PHASE) 

.6.1  CSC  INTEGRATION  AND  TESTING 

.6.2  CSCI  TESTING 

.6.3  SYSTEM  INTEGRATION  AND  TESTING 

2.7  DEVELOPMENTAL  TEST  AND  EVALUATION  (DT&E) 

2.8  FACTORY  ACCEPTANCE  TESTING 

3 . 0  OPERATIONAL  SUPPORT 

3.1  OPERATIONAL  TRANSITION 

3.1.1  OPERATIONAL  TEST  AND  EVALUATION /INTEGRATION  TESTING 

3.1.2  OPERATIONAL  TEST  AND  EVALUATION /SHAKEDOWN  TESTING 

3.1.3  PRODUCTION  ACCEPTANCE  TEST  AND  EVALUATION 

3.1.4  SITE  FIELD  SHAKEDOWN  TEST  AND  EVALUATION 

3.2  POST  DEPLOYMENT  SUPPORT 


NAS  SYSTEM  LIFE  CYCLE 
Figure  1-1 
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3.0  FINDINGS 


This  section  documents  the  findings  from  ,.e  documentation  reviews 
and  the  interviews. 


3 . 1  DOCUMENTATION  REVIEW 


3.1.1  Summary 

The  FAA  has  in  place  today  a  baseline  set  of  standards  and  orders 
which  reflect  the  current  policy  and  process  for  software  life 
cycle  management.  This  set  of  orders  and  standards  must  all  work 
together  as  an  integrated  process  and  policy.  Further,  whenever 
new  standards  or  orders  are  developed,  they  must  be  totally 
integrated  into  the  existing  set. 

This  set  includes  as  a  foundation  the  following  standards  and 
orders : 

FAA-STD-005 
FAA-STD-018 
FAA-STD-02 1 
FAA-STD-024 
FAA-STD-025 
FAA-STD-026 

Along  with  these,  a  new  Action  Notice  which  select**  Ada  as  the 
language  of  choice  for  the  FAA  has  been  approved. 

The  results  of  the  documentation  review  are  summari*.^  •  as  follows: 

(1)  In  general,  there  is  broad  but  not  specific  agreement  on  the 
definition  of  the  FAA  NAS  acquisition  life  cycle  within  the 
documents  reviewed.  The  terminology  used  within  the  standards 
and  orders  is  not  consistent.  Generally,  terms  and  products 
are  defined  but  on  occasion  terms  are  used  which  are  not 
defined. 

(2)  Some  phases  of  the  NAS  System  Life  Cycle  are  not  addressed  by 
any  of  the  reviewed  or  identified  guidelines  (see  Appendix  D)  . 
These  phases  include,  but  are  not  limited  to,  the  Concept 
Definition  and  /erification  phase,  transitioning  from 
Requirements  Definition  into  Acquisition  and  from  Acquisition 
into  Operational  Support. 

(3)  Certain  software  engineering  topics  are  not  addressed  by  any 
of  the  reviewed  FAA  standards  or  orders.  Examples  include 
Commercial  Of f-the-Shelf  Software  (COTS),  software  metrics, 
software  management,  software  reuse,  documentation  management 


FAA  Order  1800.8 
FAA  Order  1810.1 
FAA  Order  1810.2 
FAA  Order  1810.4 
FAA  Order  4630.9 
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prototyping,  and  software  risk  assessment  procedures  and 
methods .Metrics  were  specifically  mentioned  in  the  ISSS  Task 
Force  Report  (DOT/FAA-17)  as  and  area  requiring  attention, 
"...a  set  of  metrics .. .must  be  developed,  documented  and  used 
to  unambiguously  measure  progress  on  a  build  by  build  basis.” 

(4)  The  NAS-SS-1000  includes  a  list  of  applicable  standards  and 
orders  for  all  NAS  F&E  projects.  This  implies  that  R,  E&D  and 
the  maintenance  organizations  are  excluded  from  this 
requirement.  This  approach  does  not  support  a  full  NAS  life 
cycle  view  of  product  development  within  the  FAA  and 
contributes  to  the  compartmentalization  which  leads  to  lack  of 
communication  and  inefficient  development  of  software  products 
for  the  NAS. 

(5)  Since  the  FAA  has  adopted  DoD-STD-2167A  as  the  basis  for  FAA- 
STD-026,  all  other  military  documents  which  are  referenced  by 
FAA  standards  and  by  2167A  itself  may  be  out  of  date  and  must 
be  reviewed  to  assess  the  impact  of  this  situation  on  the 
guidelines  in  use  by  the  FAA.  Review  of  these  military 
documents  is  outside  the  scope  of  this  task. 

(6)  A  number  of  the  FAA  Standards  and  Orders  do  not  provide  enough 
detailed  and  specific  information  to  allow  a  straightforward 
and  consistent  implementation  of  the  stated  procedures. 
Specific  examples  are  listed  in  Appendix  B  and  are  summarized 
below  as  lacking  in  the  following  areas: 

(a)  Deliverables  -  The  standards/  orders  do  not  completely 
specify  the  deliverables  that  must  be  generated  in  accordance 
with  the  procedures.  For  those  deliverables  that  are 
identified,  only  minimal  information  is  provided  concerning 
the  format  or  content  of  the  items. 

(b)  Timeframe /Duration  -  The  standards/orders  do  not  provide 
sufficient  information  concerning  the  timeframe  and  duration 
of  the  activities  and  deliverables  that  are  addressed  in  the 
procedures . 

(c)  Activity  Details  -  The  standards/orders  do  not  provide 
enough  detailed  procedural  information  to  properly  support 
measurement  and  assurance  of  conformance  to  procedures.  The 
procedures  use  activity  descriptions  such  as  "coordinate”, 
"notify",  and  "advise"  while  providing  no  definitions  of  what 
these  terms  mean  in  the  context  of  the  standard/order. 

(7)  The  standards/orders  are  not  consistent  with  respect  to  the 
treatment  of  firmware .  Some  of  them  state  that  firmware  is  to 
be  treated  as  software,  but  most  of  them  do  not  address  the 
issue  at  all. 

(8)  "The  FAA' s  maintenance  planning  tool  is  the  national  Airspace 
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Integrated  Logistics  Support  (NAILS) .  NAILS  uses  Military 
Standard  1388  (1A  and  2A)  to  provide  Logistic  Support  Analysis 
(LSA) .  Military  Standard  1388  and  NAILS  do  not  presently 
handle  software,  accommodate  software  support  data  or  enable 
software  support  to  be  analyzed."  (WARREN89) 

(9)  Ada  has  been  adopted  by  the  FAA  as  the  preferred  language 
while  allowances  are  made  for  using  other  languages  when 
appropriate.  Language  standards  which  cover  each  language 
allowed  should  be  put  in  place  by  the  FAA. 


3.1.2  Detailed  Findings 

Appendix  B  lists  the  details  of  the  document  reviews  by  TPI  and 
itemizes  any  issues  which  require  attention  by  the  FAA. 


3.2  INTERVIEWS 


3.2.1  Summary 

For  this  task,  31  interviews  were  conducted,  and  a  survey  was 
conducted  to  determine  the  familiarity  of  the  interviewees  with 
various  documents.  The  details  of  the  survey  results  are  contained 
in  Appendix  H.  A  summary  of  the  survey's  findings  is  described 
below. 

(1)  Of  the  81  documents  presented  to  the  interviewees,  FAA  Order 
1810.4b  FAA  NAS  Test  and  Evaluation  Program  received  the 
highest  level  of  recognition.  The  second  highest  level  of 
recognition  was  DoD-STD-2l67A  Defense  System  Software 
Development . 

(2)  Of  the  81  documents,  one  (1.2%  of  all  documents)  received  a 
score  of  zero,  which  indicated  a  lack  of  knowledge  about  the 
existence  of  the  document.  This  document  was  FAA  Order 
1370.53  Uniform  Document  Standards. 

(3)  Of  the  81  documents,  9  (11.1%  of  all  documents)  received  a 
score  of  one,  which  indicated  that  the  document  is  known  to 
exist  but  its  usage  could  not  be  explained. 

(4)  Based  on  the  above  results,  a  total  of  10  (12.3%  of  all 

documents)  in  the  survey  received  either  no  recognition  or  lew 
recognition  scores. 


Gex'ieral  observations  based  on  the  interviews  are  itemized  below. 
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(1) 


Of  the  three  major  life  cycle  phases  (Requirements  Definition, 
Acquisition,  and  Operational  Support) ,  the  Requirements 
Definition  phase  is  handled  the  poorest  within  the  FAA.  Lack 
of  adequate  requirements  definition  was  stated  as  the  greatest 
influence  on  the  success  or  lack  of  success  during  all 
subsequent  phases.  Few  guidelines  exist  in  the  form  of 
standards  or  orders  which  address  this  phase.  For  the  Concept 
Definition  and  Validation  phase,  no  FAA  guidelines  are  known 
to  exist. 

The  Operational  Support  phase  was  said  to  be  the  best  phase 
because  it  "cleaned  up  the  messes"  and  did  not  allow  a  system 
to  be  deployed  which  did  not  meet  requirements  (i.e.  needs) 
even  though  it  met  specifications.  This  is  more  evidence  of 
a  problem  with  the  requirements  definition  process.  An 
inadequate  job  of  requirements  definition  leads  to  rejections 
during  testing  because  "it  wasn't  what  was  really  wanted." 
Having  the  vendor  build  the  v:rong  system,  even  correctly,  is 
very  expensive  for  the  FAA.  These  problems  should  be  caught 
much  earlier. 

(2)  The  transitions  from  Requirements  to  Acquisition  and  from 
Acquisition  to  Operational  Support  were  also  mentioned  as 
causes  for  problems.  The  transitions  are  not  well  defined  and 
no  formal  guidelines  exist  to  help  in  this  process.  A  draft 
action  notice  has  been  prepared  by  the  Software  Integration 
Working  Group  (SIWG)  which  addresses  parts  of  this  area. 
Transition  is  also  addressed  to  some  extent  in  FAA  Order 
1800.8. 

In  general,  the  mechanism  used  to  communicate  and  to  obtain 
agreements  about  the  process  during  the  Acquisition  to 
Operational  Support  transition  is  a  Memorandum  of 
Understanding.  A  contributing  factor  to  the  difficulties  with 
the  transitions  is  that  different  organizational  groups  are 
involved.  Also  the  requirements  tend  to  be  readdressed  at  the 
Acquisition  to  Operational  Support  transition  point  even 
though  prior  agreements  and  sign-offs  occurred  during  the 
Program  Definition  phase. 

(3)  One  of  the  points  that  was  mentioned  repeatedly  during  the 
interviews  was  the  lack  of  enough  skilled  and  up  to  date 
software  engineers  within  the  FAA.  This  appears  to  be  the 
case  throughout  all  phases  of  the  life  cycle  and  at  all  career 
levels.  A  lack  of  well  trained  program  managers  with  software 
background  or  knowledge  was  also  cited  as  a  problem  for  the 
FAA. 

(4)  Interpretation  and  tailoring  of  the  standards  and  orders  has 
been  difficult  for  the  FAA.  Many  of  the  standards  now 
available  are  new  to  the  FAA,  especially  DoD-STD-2167A,  and 
expertise  has  not  yet  been  developed  within  the  FAA.  DoD- 
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STD-2 167A  was  cited  over  and  over  as  having  problems  when  used 
within  the  FAA  environment.  The  FAA  views  2167A  as  adequately 
addressing  the  Acquisition  phase,  but  2167A  does  not  provide 
enough  support  for  the  Post  Deployment  Support  activities  for 
the  FAA. 

(5)  The  guidelines  do  not  stand  alone  and  do  not  have  adequate 
instructions  for  interpretation  and  use.  In  general, 
tailoring  guidelines  are  missing.  As  illustrated  in  Appendix 
A,  most  of  the  guidelines  reference  several  other  guidelines 
and  while  not  shown,  these  may  reference  several  more.  The 
lack  of  tailoring  guidelines  was  reported  several  times  as 
being  a  problem  for  the  FAA. 

(6)  The  'culture'  within  the  FAA  was  mentioned  during  the 
interviews  as  "getting  in  the  way  of  success."  This  may  be 
interpreted  to  mean  that  organizational  divisions  and 
different  perspectives  make  agreements  difficult  to  achieve. 
It  also  seems  to  imply  that  the  narrow  focus  of  each  group, 
whether  engineering  or  air  traffic,  inhibits  cooperation  and 
accommodation . 

(7)  In  general,  the  personnel  do  not  know  how  to  get  copies  of  the 
guidelines  or  what  guidelines  exist.  There  may  be  guidelines 
which  are  applicable  to  their  area  but  they  are  unknown.  In 
some  cases,  the  people  rely  upon  their  manager  to  give 
guidance  with  respect  to  standards  which  are  applicable. 

(8)  Another  issue  which  came  up  during  the  interviews  was  the 
perceived  lack  of  emphasis  on  minimizing  life  cycle  costs  for 
projects.  Each  group  appears  to  only  concern  itself  with  its 
own  phase  of  the  project.  There  are  no  apparent  incentives  to 
encourage  a  life  cycle  view  of  costs  by  all  organizations. 

(9)  Some  new  policies  or  standards  are  currently  under 
consideration.  Examples  of  these  include  a  possible  set  of 
standards  which  specify  the  minimum  level  of  qualifications 
for  a  vendor  who  provides  training  to  the  FAA  and  guidelines 
for  COTS  within  the  NAS. 


3.2.2  Detailed  Findings 

Appendix  C  lists  the  details  of  the  interviews  and  provides  a 
summary  of  the  problems  addressed  and  the  answers  received. 
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4.0 


CONCLUSIONS  AND  RECOMMENDATIONS 


4 . 1  CONCLUSIONS 

The  FAA  has  addressed  the  NAS  software  acquisition  problems  over 
the  years  and  many  guidelines  exist  as  a  result  of  these  efforts 
which  are  valuable  and  which  provide  the  needed  support  to  the  FAA 
Program  Managers  and  Software  Managers.  The  FAA  Order  1800. 8f  is 
especially  worth  noting  for  its  completeness  and  detailed 
guidelines  for  Configuration  Management.  Basically,  there  are  many 
noteworthy  guidelines  which  need  updating  in  order  to  become  more 
useful  in  today's  environment. 

A  NAS  software  acquisition  process  is  in  place  within  the  FAA  and 
this  task  has  attempted  to  clarify  that  process  and  to  evaluate 
whether  or  not  the  process  is  adequate  and  effective. 

Many  of  the  pieces  for  an  overall  NAS  software  acquisition  policy 
and  guidelines  exist  and  what  is  needed  is  the  thread  to  pull  them 
all  together  into  a  cohesive  strategic  plan.  As  a  result  of  the 
document  reviews  and  the  interviews  the  following  conclusions  were 
developed.  They  are  not  listed  with  regard  to  any  priority. 

(1)  The  Requirements  Definition  phase  of  the  life  cycle  is 
inadequately  performed  within  the  FAA.  Clearly  a  serious 
problem,  not  unique  to  the  FAA,  the  lack  of  a  sound  approach 
to  requirements  determination/specification  is  a  source  of 
life  cycle  cost  expenses  to  the  FAA  due  to  the  large  amount  of 
rework  required  to  develop  a  system  acceptable  to  the  customer 
(AT)  .  This  phase  contributes  directly  to  the  success  or 
failure  of  NAS  programs  and  must  be  given  priority  attention 
in  terms  of  resources,  including  equipment,  personnel  and 
budgets.  There  is  no  evidence  that  incentives  exist  which 
would  encourage  AT  to  define  needs  early  and  adequately. 

(2)  Requirements  are  a  moving  target  within  the  air  traffic  domain 
and  this  is  an  accepted  way  of  life  within  the  FAA  to  almost 
everyone.  However,  it  is  not  accepted  by  the  Program  Managers 
who  are  attempting  to  finish  a  program  on  schedule  and  within 
budget  against  the  SOW  and  Requirement  Specifications 
initially  bid  on  by  the  contractors. 

(3)  There  is  no  overall  software  engineering  policy  statement 
within  the  FAA  which  provides  the  system  level  guidance  for 
all  other  software  engineering  activities  within  the  FAA. 
Such  a  policy  statement  should  address  critical  areas  such  as 
organizational  commitments  and  accountability  and  containment 
of  cost  along  with  specific  software  technology  and  management 
guidelines . 
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(4)  A  consistent  system-level  definition  of  the  NAS  System  Life 
Cycle  is  missing  around  which  the  FAA  can  structure  its 
policies,  procedures,  standards  and  orders. 

(5)  A  glossary  of  terms  specific  to  the  FAA  environment,  life 
cycle  and  business  methods  is  not  available. 

(6)  Centralized  guidelines  missing  within  the  FAA  for  several 
software  engineering  topics.  The  matrix  in  Appendix  D 
illustrates  which  software  topics  are  not  addressed  by 
guidelines.  Examples  include  COTS,  software  risk  analysis  and 
risk  analysis  and  risk  management,  software  metrics,  and 
software  management.  COTS  guidelines  have  become  especially 
crucial  since  current  NAS  projects,  such  as  AAS,  have  ar. 
important  COTS  content. 

(7)  A  centralized  guideline  for  evaluation  of  vendor  standards  for 
software  design  and  code  is  not  currently  available  within  the 
FAA.  The  FAA  requires  guidelines  as  to  what  is  a  minimum 
acceptable  standard  when  evaluating  a  vendor's  proposed 
software  design  and  code  standards.  In  general,  guidelines 
are  not  available  which  address  software  technology  and  its 
management  in  this  rapidly  changing  environment. 

(8)  Some  transitions  between  phases  are  addressed  in  FAA  Order 
1800.8.  However,  the  transition  from  one  phase  to  another  in 
the  life  cycle  was  mentioned  as  a  problem  area  during  the 
interviews.  It  is  clearly  recognized  by  the  FAA  as  such  and 
steps  are  being  taken  to  address  it.  There  is  an  Action 
Notice  which  addresses  the  hand-off  process  from  acquisition 
to  operational  support  which  is  a  start  in  addressing  some  of 
the  issues. 

(9)  The  use  of  DoD-STD-2167  (and  now  FAA-STD-026/DoD-STD-2167A> 
has  been  difficult  and  the  need  for  expertise  in  these 
standards  has  been  raised  several  times  during  the  interviews. 
There  is  general  lack  of  understanding  within  the  FAA 
concerning  DOD-STD-2167A,  especially  in  the  area  of  tailoring. 
Additionally,  it  is  an  ineffective  mechanism  for 
maintenance/change  specification  since:  (a)  the  customer  (AT) 
does  not  understand  it;  (b)  its  organization  hinders  a  "clean" 
presentation  of  the  proposed  changes;  (c)  it  does  not  specify 
MMls  well;  and  (d)  it  lacks  a  Computer  Program  Functional 
Specification  (CPFS) . 

(10)  There  is  an  insufficient  number  of  strong  software  engineering 
experts  within  the  FAA  who  could  provide  the  leadership 
required  to  manage  this  complex  technology  for  the  FAA. 
Furthermore,  there  is  a  lack  of  systems  engineers  who 
understand  both  software  and  the  air  traffic  control 
environment.  There  is  also  a  lack  of  Quality  Assurance 
personnel  trained  in  software  engineering. 
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(11)  Other  than  an  informal  network  of  contacts,  there  is  no 
mechanism  for  sharing  lessons  learned  and  for  knowing  what  is 
available  in  the  software  engineering  area  within  the  FAA. 
Software  personnel  do  not  know  where  or  how  to  get  copies  of 
the  existing  guidelines. 

(12)  There  is  a  lack  of  incentives  which  encourage  and  support  the 
notion  that  life  cycle  cost  containment  is  the  responsibility 
of  each  organization  in  the  FAA. 


4.2  RECOMMENDATIONS 

(1)  The  FAA  should  implement  new  approaches  to  defining 
requirements.  The  distinction  between  needs,  requirements, 
and  specifications  should  be  addressed  and  understood.  Needs 
are  what  AT  has,  these  needs  are  expressed  through 
requirements  definitions  and  requirements  are  documented  in 
specifications  to  be  used  by  the  builders  of  the  products. 
Part  of  the  difficulties  experienced  by  the  FAA  with 
requirements  determination  may  be  that  these  distinctions  are 
not  made  and  understood  by  either  the  user  (AT)  or  the 

builders  (engineering) .  Rather  than  expecting  AT  to  define 
their  requirements,  AT  should  be  asked  to  define  and  discuss 
their  needs  (i.e.  what  problem  are  they  trying  to  solve)  and 
then  the  system  analysts  (i.e.  engineers)  should  define  the 
requirements  that  will  satisfy  the  needs  and  then  document 
these  requirements  in  a  specification. 

At  the  lower  levels  of  requirements  determination,  such  as  the 
Man-Machine  Interfaces,  a  distinction  between  functional 
requirements  and  build-to  specifications  is  not  made  within 

the  FAA.  That  is,  the  FAA  tends  to  think  of  build-to 

specifications  as  functional  requirements.  Thus,  while 
prototyping  is  used  to  define  requirements  according  to  the 
FAA,  it  often  results  in  build-to  specifications  rather  than 
functional  specifications. 

This  distinction  is  important  when  a  program  is  defined  since 
the  contractor  has  more  flexibility  with  design  and 

implementation  when  given  functional  specifications;  this  may 
then  require  more  stringent  management  and  technical  controls 
in  the  form  of  standards  being  imposed  in  the  contract.  In 
other  words,  the  tailoring  requirements  for  a  program  are 
quite  different  when  the  program  has  functional  rather  than 
build-to  specifications.  Build-to  specifications  may  be  more 
desirable  in  the  FAA  environment  for  satisfying  some 
operational  needs. 

TPI  recommends  that  the  FAA  analyze  all  aspects  of  the  way 
requirements  are  currently  defined  including  who  defines  them, 
who  is  responsible  and  accountable  for  definition  of 
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requirements,  and  what  is  the  process  used.  Although  there  are 
guidelines  in  place  for  writing  System  Requirement 
Specifications,  they  are  incomplete  and  ineffective  as 
demonstrated  by  uncontrolled  changing  requirements  and  the 
disagreements  about  whether  requirements  have  been  met  during 
the  transition  from  acquisition  to  operational  support. 

(2)  Changing  requirements  cannot  be  prevented  within  the  FAA 
operational  domain.  As  with  any  other  type  of  change,  there 
should  be  a  conscious,  well-  reasoned  decision  to  pursue 
changes  during  the  current  acquisition,  or  alternatively  to 
defer  them  to  the  future.  "Management  guidance  should 
encourage  and  support  this  deferral  and  accept  the 
consequences  of  doing  so."  [MIL-DOC-1] 

Within  the  FAA,  changed  requirements  are  often  not  identified 
until  the  transition  from  acquisition  to  operational  support. 
To  alleviate  this  problem  of  late  identification  of  changed 
requirements,  new  approaches  to  management  during  acquisition 
need  to  be  tried  which  will  seriously  involve  the  end  users  at 
all  steps  of  the  acquisition.  While  the  end  users  are  now 
expected  to  participate  in  reviews  and  to  review  all 
documentation  during  the  acquisition  phase,  this  participation 
is  apparently  not  effective  or  not  occurring  and  needs  to  be 
studied  as  to  why  it  is  failing.  Guidelines  for  managing 
changing  requirements  may  prove  useful.  A  review  process  such 
as  that  used  for  Deployment  Readiness  Review  is  suggested. 

(3)  There  is  a  need  for  an  overall  FAA  policy  statement  with 
respect  to  NAS  Software  and  the  NAS  System  Life  Cycle.  This 
policy  statement  should  be  the  guide  for  decision  making  and 
prudent  management  of  software  within  the  FAA.  Currently  there 
are  several  individual  efforts  which  concern  software 
engineering  taking  place  in  the  FAA  and  there  is  no 
coordination  between  these  efforts  to  prevent  duplication  or 
conflict.  A  policy  to  guide  these  individual  efforts  and  to 
provide  a  vehicle  for  communication  and  coordination  is 
required . 

The  policy  statement  is  not  the  panacea  for  problems  within 
the  FAA  but  can  set  the  tone  and  system  level  guidance  for  the 
software  standards  and  orders  within  the  FAA.  Furthermore,  a 
policy  statement  will  not  make  up  for  poor  software  management 
or  for  il.l-equipped  and  poorly  trained  managers. 

The  types  of  items  which  should  be  considered  by  the  policy 
statement  include: 

(a)  A  "software  first"  approach  to  NAS  System  Acquisitions. 

That  is,  address  user  needs  and  functional  requirements 
and  the  software  approach  to  satisfying  them  first 
before  specifying  the  hardware. 
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(b)  The  role  of  state-of-the-art  software  technology  and 
software  production  techniques 

(c)  Grandfathering  existing  programs  once  under  contract. 

(d)  Introduce  the  concept  of  "reasonable  judgement”  in 

management  of  software. 

(e)  The  need  to  tailor  standards  for  each  program. 

(f)  Organizational  and  management  commitments  to  software 

engineering . 

(g)  Cost  containment. 


(4)  Establish  temporary  working  groups  which  involve  members  from 
across  the  FAA  organization.  These  working  groups  would  have 
specific  mission  charters  which  address  very  specific  and 
narrowly  focused  software  engineering  needs  within  the  FAA. 
The  objectives  of  the  working  group  should  be  well  focused  on 
specific  products  to  be  developed  by  the  working  group. 
Products  should  include: 

(a)  a  NAS  software  policy  statement, 

(b)  development  of  a  consistent  definition  of  the  NAS 
System  Life  Cycle, 

(c)  development  of  an  FAA  Glossary  of  Terms, 

(d)  new  or  revised  standards  or  orders, 

(e)  Guidelines  for  evaluation  of  vendor  standards  for 
software  design  and  production. 

(f)  Guidelines  for  conduct  of  program  reviews. 

Once  the  product  is  delivered  and  has  gone  through  an 
acceptance  process,  the  working  group  should  be  disbanded  or 
assigned  another  task. 

(5)  There  is  a  scarcity  of  systems  engineers  with  software 
credentials,  a  scarcity  of  software  engineers  with  system 
perspectives,  and  few  of  either  with  air  traffic 
understanding.  Thus  TPI  recommends  the  establishment  of  a 
system  level  software  engineering  group  with  expertise 
available  to  work  with  the  FAA  Program  Managers  and  Software 
Managers.  This  group  of  software/systems/air  traffic  advisors 
can  observe  many  programs  over  a  relatively  short  time  span, 
enjoy  a  rapid  learning  curve,  and  apply  lessons  learned 
immediately . 
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This  group  should  be  available  as  in-house  consultants  to  any 
organization  within  the  FAA  which  requires  assistance  with 
systems  and  software  engineering  technology.  This  group 
should  be  staffed  by  people  with  strong  qualifications  in 
systems,  software,  and  air  traffic  control.  Over  time,  these 
people  would  become  well  versed  in  each  others  disciplines  and 
provide  the  much  needed  expertise  across  all  of  these 
disciplines . 

(6)  Establish  better  communications  vehicles  for  sharing  "lessons 
learned"  and  for  continuing  education  of  the  software 
management  in  the  FAA.  Some  possibilities  include  an  on-line 
conferencing  system,  an  on-line  database  of  all  standards, 
orders,  etc.,  an  on-line  database  of  software  topics  pointing 
to  relevant  guidelines,  quarterly  presentations  of  updated 
guidelines  and  quarterly  "lessons  learned”  brown  bag  sessions. 

(7)  Provide  a  software  managers  handbook.  This  handbook  should 
include  the  NAS  Software  Acquisition  policy  statement.  This 
handbook  should  cover  the  what,  where,  when,  and  who  of 
software  management  within  the  FAA: 

(a)  What  guidelines  the  managers  need, 

(b)  Where  to  get  the  guidelines, 

(c)  When  to  apply  the  guidelines, 

(d)  Who  to  see  for  assistance  with  the  guidelines. 

The  guidelines  themselves  should  provide  the  'how'  part  of  the 
software  managers  handbook. 

(8)  Prepare  a  technology  transfer  plan  for  software  engineering 
within  the  FAA.  Address  the  management  of  technology  change 
and  the  mechanisms  for  keeping  personnel  current,  as  well  as 
related  topics. 

(9)  One  or  two  pilot  projects  are  recommended  by  TPI  as  a  way  to 
apply  some  of  the  new  approaches  to  software  engineering 
within  the  FAA.  That  is,  bring  together  the  expert  group 
(system,  software,  air  traffic)  to  tailor  the  guidelines  for 
a  project.  Try  other  approaches  to  requirement  definition  and 
software  management.  Other  suggestions  should  be  discussed 
and  applied  as  found  appropriate  by  the  FAA. 
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5.0  PROPOSED  ACTION  PLAN 

The  action  plan  proposed  by  TPI  is  presented  in  this  section.  The 
action  plan  is  the  result  of  all  document  reviews,  personnel 
interviews,  and  feedback  received  on  the  earlier  TPI  reports.  The 
plan  is  divided  into  near,  mid,  and  long-term  tasks.  The  near-term 
tasks  are  to  be  accomplished  within  the  next  one  to  three  months. 
The  mid-term  tasks  are  to  be  accomplished  within  the  next  three  to 
twelve  months,  and  the  long-term  tasks  are  to  be  completed  beyond 
twelve  months  from  now.  The  phased-in  approach  of  the  action  plan 
supports  the  incremental  attainment  of  the  goals  identified  in 
Section  1.2. 


5 . 1  NEAR-TERM  TASKS 

In  general,  the  near-term  tasks  are  intended  to  fill  in  gaps  which 
are  immediate  in  nature  and  which  further  define  the  software 
engineering  process  within  the  NAS  System  Life  Cycle.  Attainment 
of  these  objectives  then  enables  the  mid-term  tasks  to  commence. 

(1)  Give  briefings  on  this  work  (Taskll)  to  all  of  the  FAA 
organizations  who  have  participated. 

(2)  Develop  a  "strawman"  FAA  Software  Policy  statement  and 
transition  this  into  an  official  policy  statement. 

(3)  Develop  quick  reference  charts  for  use  by  the  FAA  software 
community  showing  the  life  cycle  and  applicable  standards  and 
orders . 

(4)  Develop  a  database  of  NAS  System  Life  Cycle  vs  Guidelines 
Matrix . 

(5)  Develop  a  strategic  Software  Process  Improvement  Plan  for  the 
FAA  including  plans  for: 

a.  Identifying  and  resolving  conflicts  among  existing 
standards, 

b.  Creating  a  framework  for  practical  use  and  tailoring  of 
2167A, 

c.  Creating  a  software  engineering  training  curriculum, 

d.  Transitioning  development  among  phases  (e.g.,  entry 
and  exit  criteria,  traceability  relationships, 
consistency/completeness  criteria) , 

e.  Expanding  Action  Notice  1370.9, 
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f.  Providing  guidelines  for  dealing  with  COTS  software, 

g.  Assessing  current  FAA  software  engineering  skills, 

h.  Promoting  effective  requirements  development, 

i.  Addressing  Technology  Transfer  issues. 


5.2  MID-TERM  TASKS 

The  mid-term  tasks  continue  the  software  engineering  process 
definition  in  more  detail  and  implement  some  of  the  new  processes. 

(1)  Implement  the  FAA  Software  Process  Improvement  Plan.  Based 
upon  work  completed  during  Task  11,  implementation  is  likely 
to  include  the  tasks  discussed  below. 

(2)  Expand  the  Action  Notice  1370.9  to  address  all  of  the  issues 
of  concern  with  respect  to  Ada: 

(a)  What  systems  should  be  excluded  from  the  Ada 
requirements,  if  any, 

(b)  What  Ada  coding  standards  are  needed, 

(c)  What  Ada  metrics  are  appropriate, 

(d)  Address  any  issues  with  using  other  languages  with  Ada, 

(e)  Address  Ada  and  the  R, E&D  activities, 

(f)  Address  the  Ada  9X  impact. 

(3)  Develop  a  Software  Managers/Engineers  Handbook;  develop  an 
outline  with  the  initial  emphasis  on  what  guidelines  are 
needed,  where  to  get  the  guidelines,  when  to  apply  the 
guidelines,  and  who  to  see  for  assistance  with  the  guidelines. 

(4)  Update  the  existing  guidelines  to  reflect  the  current  FAA 
organization  and  current  technical  terminology.  Incorporate 
fixes  for  the  problems  indicated  in  Appendix  C  of  this  report. 

(5)  Develop  a  NAS  System  Life  Cycle  (expanded  version)  with  the 
addition  of  roles  information;  indicate  which  FAA 
organizations  have  responsibilities  at  each  phase  of  the  life 
cycle  and  for  which  products  and  activities  they  are 
responsible . 

(6)  Develop  and  offer  a  software  engineering  training  curriculum 
for  management  and  technical  software  engineering  personnel  in 
the  FAA,  including  such  topics  as: 
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(a)  Use  of  Software  Managers/Engineers  Handbook, 

(b)  Software  Engineering  Environments, 

(c)  Software  reuse, 

(d)  Software  risk  management, 

(e)  Software  metrics, 

(f)  Software  Project  Management, 

(g)  Software  Design  Methods, 

(h)  Style  for  the  Professional  Ada  Programmer, 

(i)  Conducting  Effective  program  reviews, 

(j)  Conducting  an  assessment  of  vendor  software  development 
processes, 

(k)  Tailoring  of  guidelines,  especially  2167a, 

(l)  Effective  Software  Quality  Assurance. 

(7)  Develop  missing  guidelines  for: 

(a)  Conducting  PDRs  and  CDRs, 

(b)  Evaluation  of  vendors  standards  for  software  design  and 
code, 

(c)  Risk  Management, 

(d)  Tailoring  of  the  guidelines,  especially  2167a. 

(8)  Initiate  investigation  of  what  the  DoD  tailoring  activities 
are  with  respect  to  DoD-STD-2167A  and  DoD-HNBK-287 .  DoD  has 
an  automated  tool  for  tailoring  of  2167A,  and  the  FAA  should 
investigate  the  feasibility  of  bring  that  tool  into  the  FAA 
environment.  The  FAA  was  a  test  site  for  this  tool,  but  it 
does  not  appear  to  be  in  use  by  the  FAA. 

(9)  Develop  a  new  approach  for  managing  changing  requirements; 
this  would  first  involve  understanding  the  current  approach  in 
terms  of  who  defines  requirements,  who  is  accountable  for 
requirements  definition  and  who  controls  budgets  which  are 
affected  by  changes  to  requirements. 

(10)  Investigate  the  current  practice  within  the  FAA  with  respect 
to  the  use  of  software  metrics  for  both  management  and 
technical  control  and  tracking  of  software  development  for  NAS 
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systems.  Select  specific  projects  now  in  progress  as  samples 
to  be  used  in  this  assessment.  Compare  these  projects  against 
a  set  of  software  metrics  which  are  considered  valid  by 
current  software  engineering  practitioners. 


5.3  LONG-TERM  TASKS 

The  long-term  tasks  fully  implement  and  provide  evaluation  of  the 

new  guidelines  and  the  new  software  engineering  processes. 

(1)  Establish  one  or  two  pilot  projects  for  application  of  the  new 
software  engineering  approaches  and  guidelines  and  assess 
impact  on  such  projects. 

(2)  Develop  an  on-line  database  of  NAS  System  Life  Cycle  items 
versus  the  applicable  guidelines  for  these  items.  This  is  to 
automate  the  matrix  in  Appendix  D  and  enable  rapid  retrieval 
of  this  information  by  the  software  community  in  order  to 
assist  them  with  their  technical  or  managerial  tasks. 

(3)  Develop  an  FAA  specific  glossary  of  terms  to  supplement  the 
IEEE  STD  729  which  is  currently  in  use  by  the  FAA. 

(4)  Define  and  put  into  practice  a  procedure  for  conducting 
independent  process  assessments  of  bidders  and  contractors  as 
part  of  the  FAA  risk  assessment  approach.  Establish  a  minimum 
set  of  requirements  with  respect  to  a  Software  Development 
Environment  for  various  types  of  contracts. 

(5)  Develop  a  NAS  project  database  which  provides  information  such 
as  what  languages  were  used,  what  standards  were  applies,  what 
documentation  was  required,  what  tailoring  was  applied  and 
other  pertinent  software  data.  This  database  could  provide 
guidelines  to  new  projects. 
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APPENDIX  C 


INTERVIEWS 


DETAILED  FINDINGS 


C . 1  INTRODUCTION 


This  section  presents  the  results  of  the  interview  processes. 
There  have  been  31  separate  interviews  for  this  task.  This  section 
presents  the  data  gathered  during  the  interview  process  along  with 
analysis  of  that  information.  Because  of  the  organization  of  this 
section  the  same  issues  appear  in  more  than  one  section.  In  many 
sections,  a  summary  of  the  issues  identified  during  the  interviews 
are  presented  along  with  the  total  number  of  subjects  raising  each 
issue . 

C . 2  PROFILES  OF  SUBJECTS 

The  subjects  interviewed  during  this  study  represented  all  aspects 
of  the  FAA' s  development  cycle.  The  subjects  had  between  one  and 
20  years  of  experience  with  the  FAA  and  were  located  at  both  FAA 
Headquarters  and  the  Technical  Center  in  Atlantic  City. 

The  following  personnel  were  interviewed: 


Chuck  Bolling 

AAT-14 

Ralph  Caprio 

ACN-310 

Jerry  Champion 

APS-410 

Ken  Clark 

APS-500 

James  Clinton 

ATR-250 

Steve  Coulombe 

ACN-130 

Loni  Czekalski 

AMC-300 

Vern  Edwards 

ADS-120 

Dennis  Emerik 

ASM-160 

Robert  Erikson 

ACN-210 

Don  Espinosa 

AHT-400 

Mary  Ann  Farrell 

LOG I CON 

John  Hamilton 

ASA-130 

Joan  Hannan 

AAP-120 

John  Horrocks 

AAP-320 

Willie  Hunter 

ASM-140 

Harry  Kane 

ASA-210 

Rick  Lay 

APS-300 

Garry  Long 

AHT-500 

Jim  Minsterl 

ASA- 6 

Jim  Monnie 

AAP-400 

Harriet  Neuman 

AAP-220 

Jacques  Press 

ACN-110 

Bill  Riehl 

ASM-160 

Steve  Smith 

ACS-320 

John  Timmerman 

ATR-210 

Gonzalo  Tornell 

ALG-410 

Robert  Ulanch 

ACD-340 

Jim  Warner 

AAF-4 

John  Wiley 

ACD-350 

Alice  Wong 

AOR-110 
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C . 3  POLICIES,  STANDARDS,  AND  ORDERS 

C.3.1  SUMMARY  OF  SURVEY  ATTITUDES  TOWARD  EXISTING  STANDARDS 

In  this  section  the  results  of  the  interviews  regarding  existing 
standards  are  documented.  Where  there  were  a  small  number  of 
opinions  concerning  a  standard,  it  was  felt  that  this  implied  that 
the  standard  does  not  pose  a  major  problem.  The  interviewees  were 
quite  emphatic  when  discussing  a  standard  that  caused  significant 
problems . 

C.3.1.1  Positive  Opinions 

The  following  standards  were  mentioned  in  a  positive  sense  by  the 
subjects.  The  number  of  subjects  that  mentioned  the  standard  in 
this  sense  is  included. 

FAA-STD-0l6a  (1) 

FAA-STD-0l8a  (2) 

FAA-STD-028  -  good,  but  hard  to  understand  (1) 
DoD-STD-2l67A  -  for  development  (1) 

DOD-STD-2167A  -  for  maintenance/replacement  of 

systems  (1) 

FAA  Order  1100  series  (1) 

FAA  Order  6100  series  (1) 

C.3.1. 2  Negative  Opinions 

The  following  standards  were  mentioned  in  a  negative  sense  by  the 
subjects.  The  number  of  subjects  that  mentioned  the  standard  in 
this  sense  is  included. 

FAA-STD-013  (1) 

FAA-STD-018  (1) 

FAA-STD-026  (3) 

FAA-STD-028  (2) 

FAA  Order  1810.4  -  too  difficult,  need  help  (1) 
DoD-STD-2167A  -  too  general  (5) 

D0D-STD-2167A  -  vendor  problems 
DoD-STD-2167A  -  for  maintenance 

DoD-STD-2167A  -  produced  too  much  documentation  (5) 
DoD-STD-2 1 67A  -  for  development;  assumes  good,  solid 

requirements  (1) 

C.3.2  CONFLICTS 

One  subject  thought  there  might  be  some  conflicts  with  some  FAA 
standards  and  DOD-STD-2167A, 

C.3.3  DISCUSSION  OF  SPECIFIC  STANDARDS 
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This  section  presents  the  issues  pertaining  specifically  to 
standards.  Included  are  the  number  of  subjects  associated  with 
each  issue. 

C.3.3.1  FAA-STD-Q1 3a  Quality  Control  Program  Requirements  (1) 

It  was  reported  that  FAA-STD-013a  has  virtually  been  superseded  by 
FAA-STDs  016a  and  018a  and  should  be  either  updated  or  eliminated. 

C.3.3.2  FAA-STD-0 1 6a  Quality  Control  Program  Requirements  (1) 

It  was  reported  that  FAA-STD-106a  is  up  to  date,  with  the  possible 
exception  of  some  terminology,  and  is  effective. 

C.3.3.3  FAA-STD-018a  Quality  Control  Program  Requirements  (3) 

Three  surveyed  had  an  opinion  on  FAA-STD-0l8a  but  there  was  not 
general  agreement.  FAA-STD-018a  was  viewed  by  some  as  more 
effective  than  DoD-STD-2168,  and  in  good  shape  and  up  to  date  (with 
the  exception  of  some  terminology) .  However-  another  subject 
thought  it  was  too  vague  and  not  detailed  enough.  No  one  described 
the  standard  as  poor,  but  some  feel  that  it  could  use  more  detail. 

C.3.3.4  FAA-STD-026  NAS  Software  Development  (3) 

FAA-STD-026  was  viewed  as  an  obstacle  and  source  of  trouble.  It 
refers  to  DoD-STD-2167A  and  includes  so  many  cross  references  that 
it  makes  tailoring  DoD-STD-2167A  difficult.  It  does  not  stand 
alone  and  provides  little  assistance  in  the  development  process. 
It  is  out  of  date  and  should  be  reworked  or  replaced. 

C.3.3.5  FAA-CTD-028  Contract  Training  Programs  (3) 

Opinion  was  divided  on  FAA-STD-028.  However,  all  thought  it  took 
a  great  deal  of  effort  to  usefully  interpret  FAA-STD-028.  In  spite 
of  this,  one  subject  thought  it  was  a  good  standard  while  the  other 
two  thought  it  was  not  good  because  of  the  difficulty  in 
interpretation.  Differences  in  interpretation  within  the  FAA  occur 
between  headquarters  and  FAA  Academy  training  personnel.  It  seems 
the  common  denominator  is  that  FAA-STD-028  is  not  clear  enough  and 
possibly  lacking  in  sufficient  detail. 

C.3.3.6  FAA  Order  1810.4b  FAA  NAS  Test  and  Evaluation  Program (1) 

One  subject  reported  trouble  with  FAA  Order  1810.4a.  It  was  stated 
that  this  order  was  not  understood  by  staff  in  both  the  Technical 
Center  and  FAA  Headquarters. 

C.3.3.7  DoD-STD-2167A  Defense  System  Software  Development  (13) 

Section  C.4  below  addresses  DOD-STD-2167A  in  detail  as  a  specific 
problem  because  it  was  the  most  widely  discussed  standard  and  the 
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one  that  presented  the  most  concern  to  the  subjects.  Based  on  the 
amount  and  strength  of  the  responses,  DoD-STD-2167A  serious 
attention  by  the  FAA. 

C.3.3.8  FAA  Order  1100  (1) 

One  subject  reported  that  the  1100  "series"  was  very  useful. 
C.3.3.9  FAA  Order  6100  Quality  Control  (1) 

One  subject  reported  that  the  6100  "series"  was  very  useful. 


C . 4  SPECIFIC  PROBLEMS 

C.4.1  DoD-STD-2 1 67 A 

DOD-STD-2167A  was  the  most  commonly  mentioned  source  of  problems 
mentioned  in  the  interview  activity.  Ten  out  of  twelve  subjects 
mentioned  DOD-STD-2167A  specifically  in  conjunction  with  some  form 
of  problem.  Specific  problems  cited  with  respect  to  D0D-STD-2167A 
were : 


Too  hard  to  use; 

Need  help  in  understanding  DOD-STD-2167A; 

Needed  help  in  tailoring,  and  FAA-STD-026  got  in  the 
way  during  this  process; 

DoD-STD-2 167 A  was  misapplied; 

DoD-STD-2 167 A  documentation  delivered,  but  not  used; 

DoD-STD-2 167A  resulted  in  too  much  documentation  to 
review; 

Although  good  for  development,  DoD-STD-2167A  documents 
are  not  suited  for  maintenance  -  too  hard  to  specify 
changes; 

DoD“STD-2167A  not  adequate  for  interactive,  real-time; 
and 

DOD-STD-2167A  loses  the  Computer  Program  Functional 
Specification  (CPFS)  concept,  and  the  CPFS  is  important 
to  the  FAA. 

The  majority  of  those  interviewed  did  not  understand  DoD-STD-2167A. 
They  had  little  or  no  training  in  this  area.  They  longed  for 
tailoring  guidelines  or  a  group  to  help  them  tailor  it.  Tailoring 
was  viewed  as  critical,  and  without  it  the  standard  would  be,  and 
was,  misapplied.  FAA-STD-026  hindered  tailoring  efforts  because  of 
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the  numerous  cross  references  to  DoD-STD-2167A  and  other  documents. 

DOD-STD-2167A  was  viewed  as  producing  a  set  of  development 
documents  rather  than  a  set  of  maintenance  documents.  In  addition 
to  tailoring,  the  second  major  problem  with  D0D-STD-2167A  was  in 
using  it  as  a  maintenance  document.  It  is  hard  to  specify  changes 
with  DoD-STD-2167A  since  they  are  often  scattered  throughout  the 
documents.  The  end-user  (AT)  cannot  understand  changes  specified 
using  DoD-STD-2l67A.  Most  interviewed  preferred  using  a  CPFS  for 
change  specification.  It  was  also  stated  that  specifying  Man- 
Machine  Interfaces  (MMls)  was  difficult  with  DoD-STD-2167A.  Since 
DoD-STD-2167A  documents  are  not  maintained  by  the  FAA,  they  get 
increasingly  out  of  date  with  time. 

However,  most  users  felt  that  with  training,  guidelines,  and  a 
maintenance  CPFS,  the  use  of  DoD-STD-2167A  would  be  effective. 

Note:  FAA-STD-018  was  viewed  as  more  effective  than  DoD-STD- 

2167A' s  companion  SoS-STD-2168  . 

C.4.2  REQUIREMENTS 

The  problem  of  Air  Traffic  (AT)  not  accurately  specifying  or 
agreeing  to  requirements  was  the  other  dominant  theme  that  surfaced 
during  the  interviews.  The  problems  cited  were: 


Cannot  get  an  firm  requirements  from  AT; 

AT  does  not  pay  attention  until  they  can  see  it; 

Lack  of  adequate  guidelines  for  requirements 
production; 

AT  does  not  follow  the  rules  with  respect  to 
requirements  specification;  and 

Projects  fail  at  the  testing  phase  due  to  vague  or 
"changed"  requirements. 

Virtually  all  interviewed,  that  had  an  opinion,  felt  that  AT,  the 
end  user/customer,  did  an  inadequate  job  in  specifying  or  agreeing 
to  requirements  for  FAA  projects.  Various  reasons  were  offered  for 
this  problem:  too  much  turnover  of  user  personnel;  lack  of 
guidelines;  too  little  time  and  budget  allocated  to  the  task; 
congressional  specified  deadlines;  and  lack  of  enforcement  of  FAA 
rules.  Some  stated  that  AT  is  incapable  of  developing 
requirements.  The  result  of  all  this  is  that  projects  fail  during 
testing  due  to  requirements  that  are  either  vague  or  no  longer  what 
the  customer  wants.  Some  felt  that  prototyping  might  help, 
especially  in  the  areas  of  MMls.  If  done  early  and  even  separately 
from  the  project's  main  contract,  it  might  serve  as  an  aid  to 
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requirements  specification,  This  might  solve  the  "AT  doesn't  know 
what  it  wants  till  it  sees  it"  problem. 


C.4.3  PROBLEMS  WITH  STANDARDS.  ORDERS .  AND  GUIDELINES 

In  addition  to  the  problems  stated  with  DoD-STD~2167A  above, 
several  other  problems  with  standards,  orders,  and  guidelines 
surfaced  during  the  interviews.  Specifically  mentioned  were: 

DoD-STD-2167A  -  addressed  separately  above;  (13) 

A  lack  of  guidelines  for  dealing  with  COTS  software; 
(5) 

The  high  cost  of  retrofitting  new  standards  to  projects 
already  underway;  (2) 

Some  orders  are  out  of  date  with  respect  to  the  FAA 
organization;  (1) 

FAA  Order  1810.4  is  hard  to  understand;  (1) 

A  lack  of  coding  standards  for  contractors;  (1) 

R, E&D  organization  doesn't  follow  any  FAA  Orders  or 
Standards,  thus  mismatched  equipment  and  systems  in  the 
NAS  (1); 

Difficulty  in  keeping  up  to  date  with  FAA  standards; 
(1) 

There  are  no  policies  or  standards  with  respect  to  air 
traffic  controller  training  on  new  enhancements,  thus 
quality  varies  greatly  (1) . 

The  DoD  procurement  approach  caused  problems  with 
yetting  test  plans  too  early;  and  (1) 

Management  was  resistive  to  change  -  hard  to  get 
standards  and  orders  updated.  (1) 

After  DoD-STD-2167A,  the  most  frequently  mentioned  problem  with 
standards  was  missing  guidelines  for  dealing  with  COTS  software. 
Issues  mentioned  were:  definition  of  a  COTS  software,  how  to  deal 
with  modification  COTS  software,  and  what  documentation 
requirements  are  needed  for  COTS  software. 

Some  problems  with  the  management,  maintenance,  and  distribution  of 
standards  were  mentioned.  Some  orders  were  not  kept  up  to  date 
with  changes  in  the  FAA  organization.  Others  cited  management 
resistance  to  change  as  a  reason  for  not  keeping  standards  up  to 
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date  with  the  technology.  Some  of  the  interviewees  cited 
difficulty  in  finding  out  which  FAA  standards  exist  and  are 
applicable  to  their  tasks.  Keeping  up  to  date  with  changes  was 
also  cited  as  difficult. 

Two  areas  were  identified  where  guidelines  were  missing.  Coding 
standards  exist  within  the  FAA  but  are  not  applies  to  vendors. 
This  results  in  code  that  is  difficult  to  maintain. 

Faa  Order  1810.4  was  cited  as  difficult  and  complex. 

A  case  was  cited  with  the  DoD  procurement  policies  which  resulted 
in  test  plans  being  developed  far  too  early. 

C.4.4  FAA  ORGANIZATION 

Some  problems  with  the  FAA  organizational  staffing  and  resources 
were  noted: 

Too  compartmentalized  -  lack  of  communication  between 
groups  within  the  FAA; 

No  group  to  handle  project  interfaces  -  lack  of  overall 
system  engineering; 

Difficulties  with  lack  of  trained  software  people  in 
the  project  and  other  offices  and  difficulty  in 
maintaining  software  project  skills  in  the  project 
office; 

Lack  of  emphasis  for  minimizing  life  cycle  project 
costs; 

Lack  of  software  skills  within  the  QA  staff;  and 

Wrong  group  is  contacted  by  Program  Manager  when 
determining  requirements  for  standards  documents  on  a 
project . 

Two  issues  concerning  the  FAA' s  organization  and  staffing  profile 
showed  up.  A  lack  of  competent  software  engineering  staff  in  the 
project  office,  software  support  and  QA  function  was  specifically 
mentioned  with  some  mention  of  the  same  problem  in  most  all  other 
areas.  The  software  background  discussed  covers:  life  cycle 
software  project  management  and  software  engineering. 

Even  if  the  problem  were  solved,  it  was  believed  that  individuals 
in  these  positions  would  either  lose  these  skills  or  leave  these 
positions.  The  possibility  of  a  "rotation  shift"  was  mentioned. 
There  was  some  mention  of  training,  but  some  felt  that  hiring  in 
the  necessary  skills  wouid  be  more  effective  than  training  existing 
staff  members. 
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The  other  issue  concerns  the  life  cycle  cost  of  FAA  projects.  It 
was  mentioned  that  not  enough  concern  for  the  life  cycle  cost  of 
the  project  was  shown  in  budgeting  effort  allocations  to  the 
various  phases  of  the  development  cycle.  A  lack  of  concern  for  the 
life  cycle  cost,  especially  in  the  earlier  phases  of  the 
development  effort,  was  also  cited.  The  Faa  was  described  as  too 
compartmentalized,  which  may  correspond  to  the  cited  lack  of 
concern  for  life  cycle  cost. 

Another  issue  raised  was  the  FAA' s  apparent  approach  to  project 
management  by  letting  standards,  rather  than  direct  involvement, 
control  the  project. 

C.4.5  QUALITY  ASSURANCE  (QA) 

QA  was  mentioned,  but  no  solid  concerns  were  evident.  The  lack  of 
software  engineering  skills  by  the  QA  staff,  as  well  as  not  enough 
early  involvement  in  project,  were  both  mentioned. 

C. 5  LIFE  CYCLE  ANALYSIS 

This  section  examines  the  various  phases  of  the  NAS  life  cycle.  In 
some  cases,  references  will  be  made  to  the  life  cycle  stages 
presented  in  Figure  1-1  of  this  report. 

C.5.1  REQUIREMENTS  DETERMINATION 

C.5.1.1  Strengths 

Only  one  person  stated  that  the  requirements  definition  process 
works  well  and  then  it  was  qualified  with  the  statement  "if  the 
requirements  are  worked  out  and  an  NCP  is  generated" . 

C.5.1. 2  Weaknesses 

The  requirements  determination  phase  of  the  NAS  life  cycle  was 
unanimously  named  the  worst  phase  of  the  life  cycle  by  those 
subjects  who  had  an  opinion.  The  major  problems  that  were  cited 
are : 


A  general  lack  of  standards  defining  the  requirements 
determination  phase; 

Little  or  no  participation  by  AT  during  the 
requirements  determination  phase; 

Not  enough  early  involvement  during  the  requirements 
phase  by  QA  and  maintenance;  and 

ATO  reacts  to  daily  situations  which  keeps  changing  the 
requirements; 
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The  real  air  traffic  controllers  are  not  defining  the 
requirements; 

Not  enough  time  allocated  for  a  thorough  job  of 
requirements  determination. 

Most  of  the  subjects  put  the  blame  on  AT  for  the  problems  in  this 
phase.  It  was  stated  that  AT  does  not  obey  the  rules  and  lets 
engineering  develop  the  requirements.  Then  AT  approves  them 
without  adequate  review.  The  other  approach  is  that  AT  states  very 
general  and  vague  requirements.  Finally,  when  AT  can  "see"  the 
product  (typically  during  testing  during  phase  2.6,  DT&E) .  it  then 
defines  the  real  requirements  for  the  project. 

No  one  denied  that  the  requirements  definition  phase  is  very 
difficult . 

There  seems  to  be  no  clear  transition  from  phase  1  to  phase  2, 
which  allows  poor  requirements  to  leak  through  to  the  next  phase. 
One  subject  suggested  some  form  of  Procurement  Readiness  Review  to 
help  verify  the  quality  of  the  requirements  statement. 

Human  factors  (MMIs)  were  cited  as  very  difficult  to  specify  in  a 
requirement  document,  prototyping  was  mentioned  as  one  useful  to 
help  this  difficult  task.  See  Section  C.7  for  more  details  on 
prototyping . 

C.5.2  ACQUISITION 

C.5.2.1  Strengths 

The  acquisition  phase  was  named  as  the  best  phase  in  the  NAS  life 
cycle.  However,  it  was  cited  as  adequate,  but  not  outstanding. 

One  subject  noted  that  programming  standards  are  not  applied  to  the 
contractors  during  this  phase  which  hurts  system  maintenance. 

Demonstrations  were  mentioned  as  a  good  way  of  determining  the 
software's  condition. 

C.5.2. 2  Weaknesses 

The  testing  guidelines  were  cited  by  several  subjects  as  being  too 
vague  and  wordy. 

One  subject  noted  the  differences  between  the  rules  and  actual 
practice  but  did  not  cite  any  specific  instances.  Other  subjects 
noted  that  often  the  guidelines  were  not  followed  for  several 
reasons.  Sometimes  the  orders  were  out  of  date  and  in  other  cases, 
standards  and  guidelines  were  simply  ignored. 

It  was  noted  that  the  standards  and  orders  applicable  during  this 
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phase  sometimes  get  out  of  date  with  respect  to  the  FAA's 
organizational  structure. 

C.5.3  OPERATIONAL  SUPPORT 

C.5.3.1  Strengths 

This  phase  appears  to  function  adequately  with  a  few  exceptions. 
Most  of  the  problems  expressed  were  a  result  of  problems  with  the 
earlier  phases  and  not  inherent  problems  with  this  phase.  One 
subject  stated  that  this  was  the  best  phase  because  it  stops 
projects  until  the  customer  is  happy  -  even  though  the  system  may 
meet  the  specifications. 

C.5.3. 2  Weaknesses 

Maintenance  was  viewed  as  hard  for  a  number  of  reasons,  most  of 
them  originating  with  the  use  of  DoD-STD-2l67A  documentation  and 
the  lack  of  a  CPFS  in  the  acquisition  phase.  It  was  noted  that, 
due  to  the  use  of  separate  organizations  to  support  hardware  and 
software,  that  maintenance  became  political  and  somewhat  difficult. 

C.5.4  PHASE  TRANSITIONS 

Both  phase  transitions  were  viewed  as  weak  points  within  the  NAS 
life  cycle.  There  are  not  clearly  understood  rules  for  defining 
these  transitions. 

Apparently,  there  are  no  standards,  orders,  etc.,  for  defining 
these  transitions,  nor  formal  procedural  methods  for  insuring  the 
adequacy  of  one  phase  before  moving  to  the  next.  It  was  suggested 
that  an  approach  similar  to  that  used  for  a  Deployment  Readiness 
Review  (DRR)  would  be  useful  in  evaluating  the  transition  from  one 
phase  to  another. 

C.5.4.1  Phase  Transition:  Requirements  Determination  (1)  to 
Acquisition  (2) 

This  transition,  although  scheduled,  never  really  occurred.  Phase 
2  begins  before  phase  1  is  completed.  Then  the  two  phases  both 
continue  until  they  both  merge  into  phase  3.  A  requirements 
determination  is  completed  before  phase  2  begins;  however,  the 
requirements  determination  is  performed  by  engineering,  not  the 
customer  (AT) .  At  does  have  to  approve  the  requirements  documents, 
but  apparently  does  not  thoroughly  review  the  document;  they  have 
not  decided  themselves  what  is  necessary.  When  testing  begins,  the 
"real  requirements"  come  out  and,  most  often,  the  real  requirements 
are  different  than  those  used  to  design  and  build  the  system. 
Obviously,  this  results  in  a  large  amount  of  costly  rework. 

There  are  rules  concerning  the  development  of  a  requirements 
document,  tut  they  are  apparently  not  followed  by  At  during  various 
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stages  of  phase  1  and  2. 

C.5.4.2  Phase  Transition;  Acquisition  (2)  to  Operational  Support 
ill 

The  pv  '£  ‘  o  3  transition,  although  not  as  much  a  problem  as  the 

phase  •  ti  2  transition,  does  cause  some  problems.  The  testing 
operation  -uided  by  a  vague  set  of  rules  and  the  exact  criteria 
for  going  '  o  chase  3  is  not  defined. 

[NOTE:  The  Jof  vare  Integration  Working  Group  (SIWG)  is  in  the 

process  of  preparing  a  "Procured  Software  Hand-off  Procedure" 
Action  Notice  which  addresses  the  transition  from  phase  2.0  to  3.0] 

C . €  DOCUMENTATION 


C . 6 . 1  PROBLEMS  ENCOUNTERED 

The  majority  of  problems  cited  concerning  documentation  were 
related  to  the  use  of  DoD-STD-2167a.  This  standard,  often 
misapplied,  results  in  documentation  that  is  not  appropriate  for 
system  maintenance  and  is  often  of  such  volume  that  it  is  not 
thoroughly  reviewed.  The  use  of  DoD-STD-2167A  also  causes  trouble 
for  the  FAA  when  .c  tries  to  tailor  the  standard.  Section  C.4  of 
this  document  contains  a  thorough  discussion  of  the  problems  with 
DOD-STD-2167A. 

C.6.2  OTHER  TOPICS 

The  Government  Printing  Office  (GPO)  often  reformats  and  publishes 
documents  produced  by  vendors.  This  is  obviously  expensive  and 
time-consuming.  It  would  be  desirable  if  the  vendors  could  deliver 
their  documentation  in  machine  readable  form  (Interleaf  was 
mentioned) . 

It  was  not  clear  how  to  handle  COTS  documentation.  The  subjects 
were  riot  clear  concerning  which  COTS  documentation  is  required  on 
a  project. 

Sometimes  it  is  cheaper  not  to  require  a  full  set  of  documentation 
but  rather  to  obtain  needed  documents  on  a  case  basis.  2167A  seems 
good  for  development  documentation  but  not  maintenance.  Since  the 
DoD-STD-2167A  documents  are  not  kept  up  to  date,  some  documentation 
must  be  maintained  in  order  to  support  maintenance.  Many  subjects 
recommended  that  the  CPFS,  used  before  DoD-STD-2167A  appeared, 
would  be  the  right  candidate  for  a  maintenance  document.  The  large 
amount  of  documentation  produced  by  DoD-STD-2167A  created  more  work 
than  the  FAA  could  handle  and  often  was  ignored  by  the  FAA. 

Many  of  these  DoD-STD-2167A  problems  can  be  solved  by  tailoring  the 
standard,  but  few  subjects  interviewed  knew  how  to  tailor  DoD-STD- 
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2167A. 


Guidelines  are  clearly  needed  to  help  the  FAA  staff  deal  with  COTS 
software  documentation.  This  was  discussed  in  the  standards 
section  above. 


C . 7  PROTOTYPING 

Prototyping  was  generally  viewed  as  a  useful  and  desirable 
activity.  It  was  most  often  cited  for  use  in  MMIs  and  once  for 
real-time  signal  processing. 

Three  advantages  to  prototyping  were  cited:  (1)  it  is  useful  for 
new  systems  where  the  activity  borders  on  R&D;  (2)  it  helps  "firm- 
up"  requirements;  and  (3)  demonstrations  of  prototypes  may  help  get 
AT  more  interested  and  involved  since  they  can  actually  see  the 
product . 

The  downside  of  prototyping  is  the  cost  of  prototyping  in  an 
accurate  simulated  environment  of  AT.  A  critical  evaluation  is 
difficult  to  do  without  a  simulated  environment. 


C . 8  METHODOLOGY  SPECIFICATION 

There  seems  to  be  no  demand  for  the  FAA  to  require  a  specific 
methodology  of  the  vendors.  There  was  concern  that  the 

documentation  and  other  standards  would  provide  sufficient 
assurance  of  this. 

C. 9  SPECIFIC  RECOMMENDATIONS  BY  THE  INTERVIEWEES 

This  section  presents  the  specific  recommendations  that  were  made 
during  the  interview  process  by  the  interviewees.  Only  the 
recommendations  explicitly  made  are  presented  here,  therefore  this 
section  is  not  a  summary  of  the  interview  process. 

The  specific  recommendations  were: 

Standards,  especially  DoD-STD-21 67A,  should  be 
tailored; 

There  should  be  a  support  group  within  the  FAA  to 
assist  in  tailoring; 

The  CPFS  concept  should  be  brought  back; 

A  draft  CPFS  should  be  done  very  early  in  the  project. 
I*,  helps  the  requirements  analysis  phase; 

Do  not  retrofit  new  guidelines  to  existing  projects; 
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Enforce  existing  standards  and  orders; 

Test  in  an  integrated  environment  in  phase  2.7  (see 
Figure  1-1),  rather  than  just  during  phase  3; 

Have  QA  and  maintenance  become  involved  early  in 
projects; 

Put  incentives  in  place  for  minimizing  life  cycle 
costing; 

Get  more  qualified  software  people  in  project  office, 
QA,  and  elsewhere; 

Develop  standards  for  dealing  with  COTS  and  non- 
developmental  software  (NDS ) ; 

Apply  FAA  coding  standards  to  contractors; 

Develop  firm  criteria  (standards)  for  moving  from  phase 
to  phase;  and 

Do  away  with  or  heavily  revise  FAA-STD-026. 

Do  more  tailoring  of  standards. 

Provide  software  engineering  concepts  training. 

Each  of  these  issues  have  been  discussed  in  other  sections  of  this 
report . 
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APPENDIX  D 


LIFE  CYCLE  ELEMENTS 
VERSOS 


GOVERNING  DOCUMENTS 


SOFTWARE  SNSINSZKIMC  ITEM 


Acgul sit  ion 


Allocated  Baseline 


Allocated  Configuration  Identification;  ACI 


Air  Traffic  Configuration  Control  Board;  AT  CCB 


oovBumo  |  covgwmjo 

IU  ORB  SB  OTHER 


Automated  Tools 


Baseline ( s) 


Case  Flies 


Change  Status  Report 


Cluster  CCBs  (see  Division  CCB) 


Code  Standards 


Commercial  Off-The-Shelf  (COTS)  Software 


Compilers 


Computer  Program  Functional  Specification 


Computer  Resource  Integrated  Support  Document 


Computer  Securlt 


Computer  Software  Component;  CSC 


Computer  Software  Configuration  Index 


Computer  Software  Configuration  Item;  CSCI 


Computer  Software  Quality  Program 


Computer  Software  Quality  Program  Plan;  CSQPP 


Computer  Software  Unit;  CSU 


Computer  System  Operator' s  Manual 


Concept  Analysis 


Concept  Definition  and  Verification 


Configuration  Audits 


Configuration  Control 


Configuration  Control  Board 


Configuration  Control  Decisions 


Configuration  Control  Procedures 


Configuration  Control  Support  Facllit 


Configuration  Management  Audits 


Configuration  Management  Plan 


Configuration  Management  Procedures 


1800. 8f  I  Form  1800-15 
1100. 134a  I  Form  1800-17 


SOFTWARE  SNOIMURrMO  ISSN 


qovrrnihg  govkrkthg 

rU  ORDER  OTHER 


Corrective  Action  Process 


Cost / Schedule  Report 


Critical  Design  Review 


CSC  Integration  and  Testin 


Database  Design  Document 


Database  Management 


Deployment  Readiness  Review;  DRR 


Design  Baseline 


Design  Configuration  Identification;  DCI 


Design  Standards 


Detailed  Design 


Developmental  Configuration 


Developmental  Test  and  Evaluation  Plan 


Developmental  Test  and  Evaluation  Procedures 


Developmental  Test  and  Evaluation  Test  Report 


Discrepancy  Reports 


Division  Configuration  Control  Board 


Documentation 


Documentation  Management 


DRR  Memorandum 


DRR  Monthly  Status  Report 


DRR  Report 


Engineering  Change  Proposals 


Engineering  Release 


FCA/PCA  Plan 


Field  Shakedown  Testin 


Firmware  Support  Manual 


Formal  Qualification  Rev 1 ew ;  FOR 


Functional  Baseline 


Functional  Conflgura tlonAud it;  FCA 


Functional  Configuration  Identification:  Fcl 


Hand-off  Package 
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SOTTWARI  ENGINEERING  ITEM 


Implementation 


Independent  OTtE  for  MSA 


Independent  Verification  and  Validation;  IVtV 


Inteqration  and  Testln 


Interface  Control  Document;  ICD 


Interface  Design  Document 


Interface  Management 


Interface  Requirements  Document;  IRD 


Interface  Requirements  Specification 


Decision  Memorandum 


Key  Decision  Point 


Logistics 


Maintenance  Engineering  Configuration  Control  Board;  ME 
CCB 


Management 


Major  System.  Acquisition 


Master  Test  Plan;  MTP 


Memorandum  of  Understanding;  MOU 


Mission  Analysis 


Monthly  Management  Review 


Monthly  Progress  Reports 


NAS  Change  Proposals 


NAS  Configuration  Control  Board;  NAS  CCB 


NAS  Life  Cycle 


NAS  System  Requirements  Specification 


Non-Development al  Software;  NDS 


GOVERNING 

OTHER 


eratlonal  Support 


Operational  Test  and  Evaluation;  OTtE 


Operational  Test  and  Evaluation  Integration  Test  Report 


rational  Test  and  Evaluation  Plan 


Operational  Test  and  Evaluation  Procedures 


Operational  Test  and  Evaluation  Shakedown  Test  Report 


eratlonal  Transition 


Physical  Configuration  Audit;  PCA 


1800. 8f 


MIL-STD-1521B 


Page  D-3 


sornonz  zrsimdrimo  xtsn 

Portable  Software 

Portability 

Post  Deployment  Support 

Preliminary  Deslqn 

Problem/Chanqe  Report 

Problem  Technical  Report 

Problem  Trackinq  and  Reportinq 

Procurement  Request 

Product  Baseline 

Product  Configuration  Identification;  PCI 

Product  Specification 

Production  Acceptance  Test  and  Evaluation  Plan 

Production  Acceptance  Test  and  Evaluation  Procedures 

Production  Acceptance  Test  and  Evaluation  Test  Report 

Proqram  Authorizations;  PA 

Proqram  Definition 

Program  Directives;  PD 

Proqram  Management  Plan 

Program  Master  Plan;  PMP 

Program  Plan 

Program  Technical  Report;  PTR 

Programmatic  Baseline 

Prelect  Implementation  Plan 

Project  Initiation 

Project  Management  Plan 

Project  Plan 

Protot vpl nq 

Quality  Assurance 

Quality  Assurance  Report 

Quality  Control 

Quality  Control  Procedures 

Quality  Control  Proqram  Plan 

Quality  Control  System  Plan 

Quarterly  Review 
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ru  order  other 


Software  Enqlneerin 


Software  Engineering  Environment 


Software  Languages 


Software  Management 


Software  Management  Plan 


Software  Test  Environment 


Software  Test  Management 


Software  Test  Plan 


Software  Test  Procedures 


Software  Test  Reports 


Software  Tools 


Software  Top  Level  Design  Document 


ransrtion 


Software  Transition  Plan 
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Software  User's  Manual 


Source  Code 


Specification  Change  Notices 


Specification  Review  Board 


Specifications 


Statement  of  Work 


Subcontractor  Management 


Page  D-7 


tomui  XMOIMURHK  ISSN 


oovKumwc 

TAX  ORDER 


QOTXftirXttQ 

OTHKR 


Verification  Requirements  Traceability  Matrix; 
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EXPANDED  NAS  SYSTEM  LIFE  CYCLE 


EXPANDED  NAS  SYSTEM  LITE  CYCLE 


1.0  REQUIREMENTS  DETERMINATION:  (all  FAA  actlvitias) 

1.1  CONCEPT  DEFINITION  AND  VERIFICATION 

1.1.1  Products : 

System  Level  Operational  Concept  Documents 
R,  E&D  Program  Plans 
R, E&D  Project  Plans 

System  Specification,  Program  (initial) 
Functional  Specification 

1.1.2  Reviews : 

Monthly  Management  Reviews 

1.1.3  Actions : 

Concept  analyses 
Mission  analyses 
Technology  application  studies 


1.2  PROGRAM  DEFINITION  (MSA:  Requirements  Definition) 

1.2.1  Products : 

NAS  SYSTEM  LEVEL: 

NAS  Requirements  Document  (initial) 

NAS  System  Requirements  Specification  (baselined) 

NAS  Level  I  Design  (functional  baseline) 

NAS  System  Specification  (allocated  baseline) 

(also  called  NAS  Level  II  Design) 

NAS  Transition  Plan  (initial) 

(also  called  NAS  Level  III  Design) 

NAS  Interface  Requirements  Document  (baselined) 

PROGRAM  SYSTEM  LEVEL: 

Major  System  Acquisition  Candidate  Statement 
Mission  Need  Statement  (for  program  not  in  NAS  Plan) 
System  Requirements  Statement;  SRS  (initial) 

Key  Decision  Memorandum;  KDM  (initial) 

Acquisition  Paper;  AP 

Program  Directives  (for  testing) 

Clearance  Records  (T&E) 

Program  Management  Plan;  PMP 

(also  called  Program  Master  Plan, 

Project  Management  Plan  1810. Id) 
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System  Specification  (baselined) 

(also  called  Project  Specification, 

Program  Specification) 

Master  Test  Plan;  MTP  (initial) 

(includes  Verification  Requirements  Traceability  Matrix;- 

VRTM) 

Request  for  Proposal 
Work  Breakdown  Structure 
Configuration  Control  Decisions  (CCD) 

Case  Files 

Statement  of  Work  (SOW) 

NAS  Change  Proposals 
Risk  Management  Plan 
Contract  Training  Proposals 

SOFTWARE : 

Software  Acquisition  Plan 
Software  Management  Plan 

Software  Configuration  Management  Plan  (initial) 

Software  Maintenance  Plan  (initial) 

Software  Transition  Plan  (initial) 


NAS  SYSTEM  LEVEL: 

NAS  System  Requirements  Review 

NAS  System  Engineering  Configuration  Control  Board;  CCB 
Quarterly  Review 

PROGRAM  SYSTEM  LEVEL: 

Monthly  Management  Review 
Specification  Review  Board  (SRB) 

System  Engineering  CCB 

Test  Policy  &  Planning  Review  Board  (TPRB)  meeting 
(MTP  review) 

SOFTWARE: 

TBD 

NAS  SYSTEM  LEVEL: 

Designate  *  program  as  a  Major  System  Acquisition  or  not 
a  MSA 
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appoint  Program  Sponsor 
appoint  Program  Manager 
approve  Program  Manager  charter 


PROGRAM  SYSTEM  LEVEL: 

Identify  training  requirements  (for  FAA  personnel) 
Define  Operational  Test  and  Evaluation  OT&E) Integration 
and  Shakedown  requirements 
Define  Production  Acceptance  Test  and  Evaluation 
(PAT&E)  requirements 

Validate  Deployment  Readiness  Review  (DRR)  items  in  the 
solicitation  package 

SOFTWARE : 

Assess  software  technology  requirements 


r  Decision  Point  (KDP)  #1  For  MSA: 


Authorizes  program  to  proceed  to  Concept  Analysis 


EXPANDED  MAS  SYSTEM  LIFE  CYCLE 


2.0  ACQUISITION:  (contractor  activities  except  items  with  *  are 

FAA  activities) 

2.1  PROJECT  INITIATION  (MSA:  Concept  Analysis) 

2.1.1  Products : 

SYSTEM  LEVEL: 

Program  Management  Plan 
Configuration  Management  Plan  (initial) 

Quality  Control  Program  Plan;  QCPP  (initiax) 

Quality  Control  System  Plan;  QCSP  (initial* 
System/Segment  Design  Document;  SSDD  (init/.al) 

(also  called  System/Segment  Specification;  SSSj 
Risk  Management  Plan  (initial) 

Training  Plan  (initial) 

Monthly  Progress  Reports;  Cost/Schedule  Reports 
Engineering  Change  Proposals  (ECP) 

(also  called  Engineering  Change  Requests  -  018a) 
Specification  Change  Notices  (SCN) 

Configuration  Control  Decisions  (CCD) 

•System  Development  Contract 
•Quarterly  Status  Reports 
•Test  Management  Plan 

SOFTWARE : 

Software  Development  Plan  (initial) 

(includes  Software  Engineering  Environment  Plan) 
Software  Requirements  Specification  (initial) 

Interface  Requirements  Specification  (initial) 

Software  Configuration  Management  Plan  (initial) 
Computer  Software  Quality  Program  Plan;  CSQPP  (initial) 
Software  Standards  and  Procedures  Specification  (initial) 
(includes  software  design  and  code  standards) 
Traceability  Matrix;  System  Spec.,  SOW  (initial) 


2.1.2  Reviews : 

SYSTEM  LEVEL: 

Monthly  Management  Review 
System  Requirements  Review 
Program/pro ject  CCB 
•Quarterly  Reviews 

S  DC  TWARE : 
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Software  Specification  Review;  SSR 

2.1.3  Actions : 

SYSTEM  LEVEL: 

TBD 

SOFTWARE : 

Software  methodology  selection 
Software  tools  selection 
Language  selection 
Operating  System  selection 
Build  versus  buy  decisions 
Software  metrics  selection 
Software  Tools  demonstrations 
Configuration  Control  Tool 
Software  Development  Library 
Traceability  Matrix  Tool 
Problem  Tracking  and  Reporting  Tool 
Software  development  tools  (compilers,  etc) 


2.2  REQUIREMENTS  DEFINITION  (MSA:  Concept  Analysis) 

2.2.1  Products : 

SYSTEM  LEVEL: 

Configuration  Management  Plan  (baselined) 

Quality  Control  Program  Plan;  QCPP  (baselined) 

Quality  Control  System  Plan;  QCSP  (baselined) 
System/Segment  Design  Document;  SSDD  (baselined) 

(or  System/ Segment  Specification;  SSS  (baselined) ) 
Risk  Management  Plan  (baselined) 

System  Test  Plan;  STP  (initial) 

(this  is  the  Development  Test  and  Evaluation  (DT&E) 
Plan) 

Monthly  Status  Reports 
Contract  Training  Plan 
Computer  System  Diagnostic  Manual 
System  Allocation  Document 

Computer  Resource  Integrated  Support  Document;  CRISD 
(initial) 

"Quarterly  Status  Reports 
SOFTWARE: 
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Software  Development  Plan  (baselined) 

Software  Requirements  Specification  (baselined) 

Software  Configuration  Management  Plan  (baselined) 
Computer  Software  Quality  Program  Plan  (baselined) 
Software  Standards  and  Procedures  Specification 
(baselined) 

Software  Quality  Control  Procedures  (initial) 

Interface  Requirements  Specification  (baselined) 
Traceability  Matrix  (SSDD,  SRS,.  IRS) 

Software  Quality  Assurance  Reports 

Computer  Program  Functional  Specification;  CPFS  (initial) 


2.2.2  Reviews : 


SYSTEM  LEVEL: 

Monthly  Management  Review 
System  Requirements  Review;  SRR 
System  Design  Review;  SDR 
♦Quarterly  Review 

SOFTWARE : 

Software  Specification  Review;  SSR 

Software  Products  Evaluations;  SDP,  SSDD,  SRS,  IRS 


2.2.3  Actions : 

SYSTEM  LEVEL 

Submit  Engineering  Change  Proposals  (ECP) 

Configuration  Control  Board  acts  on  ECPs 
Establish  project  Problem  Tracking  and  Reporting  database 
Financial  Data  Analysis 
Training  Course  Task  Analysis 

SOFTWARE : 

Establish  project  Software  Development  Library 
Software  Metrics  Analysis 


2.3  PRELIMINARY  DESIGN  (MSA:  Concept.  Analysis) 

2.3.1  Pgp.flMCtS; 

SYSTEM  LEVEL: 
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2.3.2 


2.3.3 


Monthly  Status  Reports 
Training  Plan  (baselined) 

System  Test  Plan;  DT&E  (baselined) 
System  Test  Procedures;  DT&E  (initial) 
Training  Materials  (initial) 

Contract  Training  Plan 
♦Master  Test  Plan  (baselined) 

♦Quarterly  Status  Reports 

♦Key  Decision  Memorandum  (updated) 


SOFTWARE : 

Software  Quality  Control  Procedures  (baselined) 

Software  Test  Plan  (initial) 

(includes  Software  Test  Environment  Plan) 

Interface  Design  Document;  IDD  (initial) 

Software  Top  Level  Design  Document;  STLDD  (initial) 
Traceability  Matrix  (updated  -  STLDD) 

Software  Development  Files;  STLDD 

(also  called  Software  Development  Folders  -  018a) 
Software  Quality  Assurance  Reports 

Computer  Program  Functional  Specification;  CPFS 
(baselined) 


Reviews : 

SYSTEM  LEVEL; 

Monthly  Management  Reviews 
♦Quarterly  Review 

SwTWARE: 

Software  Product  Evaluations;  STLDD 
Preliminary  Design  Review  (PDR) 
Configuration  Management  Audits 


ftsUtfna; 

SYSTEM  LEVEL; 

Financial  Data  Analysis 

Submit  Engineering  Change  Proposals  (ECP) 

Configuration  Control  Board  acts  on  ECPs 

SOFTWARE : 
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Resource  Performance  Analysis 
Software  Metrics  Analysis 

2.3.4  KDP  #2  for  MSA: 

♦Authorizes  program  to  proceed  to  Demonstration  Phase 


2.4  DETAILED  DESIGN  (MSA:  Demonstration  Phase) 

2.4.1  Products : 

SYSTEM  LEVEL: 

Monthly  Status  Reports 
System  User's  Guide  (training) 

Course  Design  Guide  (training) 

♦Quarterly  Status  Reports 

SOFTWARE : 

Software  Test  Plan;  STP  (baselined) 

Interface  Design  Document,  Top  Level;  TLIDD  (baselined) 
Software  Top  Level  Design  Document;  STLDD  (baselined) 
Software  Test  Descriptions,  Cases  (initial) 

Software  Detailed  Design  Document;  SDDD  (initial) 
Interface  Design  Document,  Detailed;  DIDD  (initial) 
Traceability  Matrix  (updated  -  SDDD) 

Software  Quality  Assurance  Reports 

Software  Development  Files  (updated  -  SDDD,  CSC) 


2.4.2  Reviews : 

SYSTEM  LEVEL: 

Monthly  Management  Reviews 
♦Quarterly  Review 

SOFTWARE : 

Critical  Design  Reviews  (CDRs) 
Configuration  Management  Audits 


2.4.3  Actions : 

SYSTEM  LEVEL: 

Submit  Engineering  Change  Proposals  (ECP) 
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Configuration  Control  Board  acts  on  ECPs 
SOFTWARE : 

Resource  Performance  Analysis 
Software  Metrics  Analysis 
Configuration  Control;  STP,  TLIDD,  STLDD 


2.5  IMPLEMENTATION  (MSA:  Demonstration  Phase) 

2.5.1  Products : 

SYSTEM  LEVEL: 

System  Test  Procedures;  DT&E  (baselined) 

Production  Acceptance  Test  and  Evaluation  Plan;  PAT&E 
(initial) 

Monthly  Status  Reports 

♦Operational  Test  and  Evaluation  Plan;  OT&E  (initial) 
♦OT&E  Procedures  (initial) 

♦Quarterly  Status  Reports 

SOFTWARE : 

Source  Code 

Computer  System  Operator's  Manual;  CSOM 
Software  Programmer's  Manual;  SPM 
Firmware  Support  Manual;  FSM 

Software  Detailed  Design  Document;  SDDD  (baselined) 
Software  User's  Manual;  SUM  (initial) 

Software  Test  Descriptions,  procedures  (initial) 
Traceability  Matrix  (updated  -  source  code) 

Software  Quality  Assurance  Reports 
Software  Development  Files  (updated  -  CSU) 


2.5.2  Reviews : 

Monthly  Management  Reviews 
♦Quarterly  Review 


2.5.3  Actions : 


Resource  Performance  Analysis 
Software  Metrics  Analysis 

Configuration  Control;  SDDD,  DIDD,  STD,  CSU 
Submit  Engineering  Change  Proposals  (ECP) 
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Configuration  Control  Board  acts  on  ECPs 


2.6  INTEGRATION  AND  TESTING  (MSA:  Demonstration  Phase) 

2.6.1  CSC  Integration  and  Testing: 

2. 6. 1.1  Products: 


SYSTEM  LEVEL: 

Monthly  Status  Reports 
♦Quarterly  Status  Reports 

SOFTWARE : 

Software  Test  Reports 

Traceability  Matrix  (updated  -  CSC  tests) 

Software  Quality  Assurance  Reports 

Software  Development  Files  (updated  -  CSC  tests) 


2 . 6 . 1 . 2  Reviews : 

Monthly  Management  Review 
♦Quarterly  Review 

2. 6. 1.3  Actions: 


Submit  Engineering  Change  Proposals  (ECP) 
Configuration  Control  Board  acts  on  ECPs 


2.6.2  CSCI  Testing; 

2. 6. 2.1  Products: 

SYSTEM  LEVEL: 

Monthly  Status  Reports 
♦Quarterly  Status  Reports 

SOFTWARE: 

Software  Test  Reports 

Traceability  Matrix  (updated  -  CSCI  tests) 

Software  Quality  Assurance  Reports 

Software  Development  Files  (updated  -  CSCI  tests) 

2. 6. 2. 2  Reviews: 


Page  *-10 


EXPANDXD  HAS  SYSTEM  Lin  CYCLK 


Monthly  Management  Review 
•Quarterly  Review 


2. 6. 2. 3  Actions: 


Submit  Engineering  Change  Proposals  (ECP) 
Configuration  Control  Board  acts  on  ECPs 


2.6.3  System  Integration  and  Testing; 

2. 6. 3.1  Products: 

SYSTEM  LEVEL: 

Monthly  Status  Reports 
DT&E  Procedures  (baselined) 

PAT&E  Plan  (baselined) 

PAT&E  Procedures  (initial) 

•Project  Implementation  Plan 
•OT&E  Plan  (baselined) 

•OT&E  Procedures  (baselined) 

•Quarterly  Status  Reports 

•Key  Decision  Memorandum  (updated) 

SOFTWARE : 

Software  Test  Reports 

Software  User's  Manual;  SUM  (baselined) 

Software  Test  Descriptions,  procedures  (baselined) 
Training  Materials  (baselined) 

Version  Description  Documents-  (initial) 

Software  Product  Specifications  (initial) 

Traceability  Matrix  (updated  -  system  test  procedures) 
Software  Quality  Assurance  Reports 

2. 6. 3. 2  Reviews: 

Monthly  Management  Review 
•Quarterly  Review 
•Test  Readiness  Review 

2. 6. 3. 3  Actions: 


Submit  Engineering  Change  Proposals  (ECP) 
Configuration  Control  Board  acts  on  ECPs 
Functional  Configuration  Audit  (FCA) 
Physical  Configuration  Audit  (PCA) 
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2.6.4  KDP  #3  for  MSA: 

♦Authorizes  program  to  proceed  with  full  scale  development  and  limited- 
production/  return  to  Step  2.0. 


2.7  FACTORY  ACCEPTANCE  TESTING  (DT&E) 

2.7.1  Products ; 

SYSTEM  LEVEL: 

DT&E  Test  Reports 
Monthly  'Status  Reports 
♦Quarterly  Status  Report 

♦Test  Support  Memorandum  of  Understanding;  MOU  (initial) 
SOFTWARE : 

Updated  source  code 

Version  Description  Documents  (baselined) 

Software  Product  Specifications  (baselined) 

Traceability  Matrix  (updated  -  all) 

Computer  Resource  Integrated  Support  Document;  CRISD 
(initial) 

Software  Quality  Assurance  Reports 


2.1 .2  Reviews ; 

Functional  Configuration  Audit  (FCA) 
Physical  Configuration  Audit  (PCA) 
Monthly  Management  Review 
♦Quarterly  Review 


2.7.3  Asti-on?; 


Formal  Qualification  Testing  (FQT) 
Configuration  Control  Board  acts  on  ECPs 
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3.0  OPERATIONAL  SUPPORT  (contractor  and  FAX  activities) 

3.1  OPERATIONAL  TRANSITION 


3.1.1  Operational  Test  and  Evaluation/Intecration  Testing: 

3. 1.1.1  Products: 


Computer  Resource  Integrated  Support  Document; 
(baselined) 

♦Program  Directives 

’"'Memorandum  of  Understanding;  MOU  (final) 

♦OT&E  Integration  Test  Report 
♦Quarterly  Status  Reports 

♦DPR  Memorandum  (announce  DPR  Team  Meeting) 

3. 1.1. 2  Reviews: 

Monthly  Management  Review 
♦Quarterly  Review 

3. 1.1. 3  Actions: 


Initial  review  of  DRR  checklist 


3.1.2  Operational  Test  and  Evaluation/Shakedown  Testing; 

3. 1.2.1  Products: 

♦Quarterly  Status  Reports 
♦OT&E  Shakedown  Test  Reports 
♦DRR  Team  Meeting  Report 
♦DRR  Monthly  Status  Report 

3 . 1 . 2 . 2  Reviews : 

Monthly  Management  Review 
♦Quarterly  Review 
♦Deployment  Readiness  Review  (DRR) 

3. 1.2. 3  Actions: 

TBD 

3.1.3  Production  Acceptance  Test  and  Evaluation: 


CRISD 
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3. 1.3.1  Products: 

PAT&E  Test  Reports 
♦Quarterly  Status  Reports 

3 . 1 . 3 . 2  Reviews : 

Monthly  Management  Review 
♦Quarterly  Review 

3. 1.3. 3  Actions: 

TBD 

3.1.4  Site  Field  Shakedown  Test  and  Evaluation 

3. 1.4.1  Products: 

♦T&E  Reports 

♦Quarterly  Status  Reports 
♦Key  Decision  Memorandum  (updated) 
♦DRR  Team  Meeting  Report 
♦DRR  Monthly  Status  Report 

3. 1.4. 2  Reviews: 

Monthly  Management  Review 
♦Quarterly  Review 
♦Deployment  Readiness  Review  (DRR) 

3. 1.4. 3  Actions: 

System  Commissioned 


3.1.5 


for  MSA: 


♦Authorizes  program  to  proceed  with  full  production,  installation 
and  operation  of  the  system. 
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3.2  POST  DEPLOYMENT  SUPPORT 

3.2.1  Products : 

Problem  Technical  Report  (PTR) 

Case  Pile 

NAS  Change  Proposals 
Updated  source  code 

Updated  documentation  (what  documentation?) 

3.2.2  Reviews : 

Monthly  Management  Review 

3.2.3  Actions : 


Live  environment  shakedown  testing 
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APPENDIX  G 
GLOSSARY /ACRONYMS 


APPENDIX  6 


GLOSSARY/ACRONYMS 


AAF 
AAP 
ACD 
AC  I 
ACN 
ADS 
AHT 
ALG 
AMC 
ASA 
ASE 


ASM 

AT 

ATC 

ATR 

CCB 

CDR 

CM 

COTS 

CPFS 

CRISD 

CSC 

CSCI 

CSOM 

CSQPP 

CSU 

DBDD 

DCI 

DID 

DOD 

DRR 

FAA 

FCA 

FCI 

FQR 

FSM 

GFS 


Associate  Administrator  for  Airway  Facilities 
Automation  Service 

Engineering,  Research  and  Development  Service 

Allocated  Configuration  Identification 

Engineering,  Test  and  Evaluation  Service 

Advanced  System  Design  Service 

Office  of  Training  and  Higher  Education 

Logistics  Service 

Management  Control  Service 

Advanced  System  Acquistion  Service 

System  Engineering  and  Program  Management 

Office 

Systems  Maintenance  Service 

Air  Traffic 

Air  Traffic  Control 

Air  Traffic  Plans  and  Requirements  Service 
Configuration  Control  Board 
Critical  Design  Review 
Configuration  Management 
Commercial  Off-the-shelf 

Computer  Program  Functional  Specification 

Computer  Resources  Integrated  Support  Document 

Computer  Software  Component 

Computer  Software  Configuration  Item 

Computer  Software  Operator' s  Manual 

Computer  Software  Quality  Program  Plan 

Computer  Software  Unit 

Database  Design  Document 

Design  Configuration  Identification 

Data  Item  Description 

Department  of  Defense 

Deployment  Readiness  Review 

Federal  Aviation  Administration 

Functional  Configuration  Audit 

Functional  Configuration  Identification 

Formal  Qualification  Review 

Firmware  Support  Manual 

Government  Furnished  Software 
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ICD 

IDD 

IRD 

IRS 

IV&V 

LSA 

MOU 

MTP 

NAS 

NDS 

OSD 

OT&E 

PCA 

PCI 

PD 

PDR 

PMP 

PTR 

QA 

QC 

SDD 

SDDD 

SDF 

SDP 

SDR 

SE 

SPM 

SRS 

SRS 

SSDD 

SSS 

SUM 

STD 

STLDD 

STP 

TP  I 

WD 

VRTM 


Interface  Control  Document 

Interface  Design  Document 

Interface  Requirements  Document 

Interface  Requirements  Specification 

Independent  Verification  and  Validation 

Logistics  Support  Analysis 

Memorandum  of  Understanding 

Master  Test  Plan 

National  Airspace  System 

Non-developmental  Software 

Operation  and  Support  Document 

Operational  Test  and  Evaluation 

Physical  Configuration  Audit 

Product  Configuration  Identification 

Program  Directives 

Preliminary  Design  Review 

Program  Master  Plan 

Program  Technical  Report 

Quality  Assurance 

Quality  Control 

Software  Design  Document 

Software  Detailed  Design  Document 

Software  Development  Files 

Software  Development  Plan 

System  Design  Review 

System  Engineering 

Software  Programmer's  Manual 

Software  Requirements  Specification 

System  Requirements  Statement 

System/Segment  Design  Document 

System/Segment  Specification 

Software  User's  Manual 

Software  Test  Description 

Software  Top  Level  Design  Document 

Software  Test  Plan 

Technology  Planning,  Incorporated 

Version  Description  Document 

Verification  Requirements  Traceability  Matrix 
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APPENDIX  B 


GUIDELINES  SURVEY  RESULTS 


This  appendix  contains  the  results  of  a  survey  conducted  during  the 
interview  process  concerning  the  familiarity  of  the  interviewees 
with  the  various  standards,  orders,  and  guidelines  used  by  the  FAA 
and  in  particular  by  the  interviewees.  Each  person  was  asked  to 
indicate  which  documents  they  had  used  or  were  familiar  with  at 
some  level  and  to  indicate  on  a  scale  from  1  to  5  their  level  of 
familiarity.  Most  familiar  is  a  5  and  indicates  that  the  person 
knows  the  document  well  enough  to  explain  its  use  to  someone  else. 
Least  familiar  is  a  1  and  indicates  they  know  of  it  but  could  not 
explain  its  usage.  A  rating  of  0  indicates  that  they  did  not  know 
of  the  existence  of  the  document. 

The  raw  results  are  presented  here  but  no  conclusions  have  been 
drawn  from  this  data  for  the  following  reasons.  The  sample  is  too 
small  to  be  of  much  significance  and  the  survey  was  not  scientific 
in  nature.  The  first  three  interviewees  were  not  asked  to  indicate 
their  familiarity  on  the  1  to  5  scale;  this  was  a  change  in  the 
interviewing  process.  Their  results  are  indicated  under  the 
'other'  column. 

If  this  data  appears  to  be  of  interest,  a  more  scientific  survey 
should  be  conducted.  The  outcomes  of  such  a  survey  would  include 
indications  of: 

a.  training  needs, 

b.  lack  of  enforcement, 

c.  old,  unused,  and  un-needed  guidelines, 

d.  guidelines  which  are  heavily  used. 

The  heavily  used  guidelines  may  provide  insight  as  to  why  some 
guidelines  are  successfully  used  and  others  are  not. 

The  columns  show  numbers  which  are  the  number  of  interviewees  who 
had  that  level  of  familiarity  with  the  guidelines.  For  example, 
the  number  4  for  DD  Form  1423  under  column  header  #5  means  that  4 
persons  indicated  a  level  5  of  familiarity  with  that  item.  The 
TOTAL  column  indicates  the  total  number  of  interviewees  who 
responded  as  knowing  that  item. 


Page  H-l 


GUIDELINES  SURVEY  RESULTS 


TITLE 

AC  00-41 

FAA  Quality  Control  System  Certification 
Program  (for  quldanee  and  Information) 

AFSC  Pamphlet  800- 
43 

Air  Force  Systems  Command  Software 

Management  Indicators 

ANSI  V32.16 

American  National  Standards  Institute 
Reference  Designations  for  Electrical  and 
Electronic  Parts 

DO  Form  1423 

Contract  Data  Requirements  List 

DD  Form  1664 

Data  Item  Description 

DoD  FAR  Supplement 
27.410-6 

DoD  5000. 19-L,  Vol. 

II  AMSDL 

DoD-HDBK-287 

A  Tailoring  Guide  for  DoD-STD-2l67A, 

Defense  System  Software  Development 

DoD-STD-2 1 67 A 

Defense  System  Software  Development 

DoD-STD-2168 


DoD-STD- 480 


DOT/FAA/ES-85/03 


FAA  Order  UOO.Ule 


FAA  Order  U00.124 


FAA  Order  HOP. 122b 


FAA  Order  HOC. 134a 


FAA  Order  1100.145b 


FAA  Order  1320.33 


FAA  Order  1320.48b 


FAA  Order  1370.52b 


FAA  Order  1370.53 


FAA  Order  1600.2 


FAA  Order  1800.40 


FAA  Order  1600. S4 


FAA  Order  1600.8 


FAA  Order  1800.25 


FAA  Order  1800.58 


Defense  System  Software  Quality  Program 


Con.f  Iqurat  ( nn  Control  Requirements 


NAS  Training  Plan 


Management  of  Air  Traffic  Control 
Automation 


AT/AF  Responsibilities  at  NAS  Computer 
Equipped  ARTCCa 


Airway  Facilities  Sector  Conf lyrratlon 


Maintenance  of  NAS  Automation  Subsystem 


Program  Technical  Report  (PTR)  Procedures 


Equipment  Modification  and  Facility 
Instruction 


Engineering  Field  Support  Sector 
Maintenance  Program  Procedures 


Information  Resources  Management 
Policies  end  Procedures 


Uniform  Document  Standards 


National  Security  Information 


Security  for  Electronically  Transmitted 
Message  i 


Security  of  FAA  Automated  Data  Processing 
Systems  and  Facilities 


_CorWTiun_l£aH_on_Secu£lit^_ 


Configuration  Control  Support  Facility 


National  Airspace  Integrated  Logistics 
Support  Policy 
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GUIDELINES  SURVEY  RESULTS 


DOCUMENT  Ni'MRFR 

TITl.F. 

FAA  Order  1800. 8£ 

National  Airspace  System  Configuration 
Management 

FAA  Order  18 1C. Id 

Malot  Systems  Acquisition  Management 

FAA  Order  1810.2 

Independent  Operational  Teat  and 

Evaluation  for  Halor  Systems  Acquisition 

FA  Order  1810.3 

Cost  Estimation  Policy  and  Procedures 

FAA  Order  1810.4b 

FAA  NAS  Teat  and  Evaluation  Program 

FAA  Order  3020. la 

Use  of  Computer-Based  Instruction 

FAA  Order  4405.15 

Reprocurement  Data  Acquisition  Policy 

FAA  Order  4453. 2a 

FAA  Quality  Control  System  Certification 
Program 

FAA  Order  463C.8 

Quality  Assurance  Policy 

FAA  Order  4630.9 

FAA  Computer  Software  Quality  Program 
Requirements 

FAA  Order  6000.10 

Airway  Facilities  Service  Maintenance 
Proqram  (inactive! 

FAA  Order  6000. 30a 

Policy  for  Maintenance  of  the  NAS 

FAA  Order  $032. 1A 

Modification  to  Ground  Facilities, 

Systems,  and  Equipment  In  the  NAS 

FAA  Order  6103,1a 

Maintenance  of  NAS  Er.Routc  Stage-A  .Mr 
Traffic  Control  System 

FAA  Order  6100.9c 

Quality  Control 

FAA  Order  6120.1a 

Facility  Modifications  to  ARTS  IJIA  Air 
Traffic  Maintained  Software 

FAA  Order  7032.2b 

Air  Traffic  Operatlonjl  Requirements 

FAA  Order  7800.20 

Proqram  Technical  Report  IPTr)  Procedure 

FAA  Order  7800. 7b 

Costlnq  for  Proqram  System  Version  Updates 

FAA  Order  7880.22* 

Identification  of  Source  Code  Chanqe 

FAA-D-2494 

Technical  Publications 

FAA-STD-002 

Engineering  Drawlnqs 

FAA-STD-0O5d 

Preparation  of  Specification  Documents 

FAA-STD-013* 

FAA-STD-0 1 6a 

FAA-STD-018a 

Computer  Software  Quality  Program 
Requirements 

FAA-STD-021a 

Configuration  Management  (contractor 
requirements) 

FAA-STD-024a 

Preparation  of  Test  and  Evaluation 

Documentation 

FAA-STD-025b 

Preparation  of  Interface  Control 

Documentation 
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DOCUMENT  NUMBER 


GUIDELINES  SURVEY  RESULTS 


FAA-STD-026 


NAS  Software  Development 


FAA-STD  028 


Contract  Training  Programs 


FAA-STO-OOO 


Preparation  of  Procurement  Request 
Packages 


FAA-STD-031 


Preparation  of  Statement  of  Work 


FAA-STD-034 


Instructions  for  the  Preparation  of 
Logistics  Support  Analysis  (LSA)  Data 


FAA-STD-035 


Commercial  Equipment,  Market  Research  for 
Preparation  of  Project  Implementation 
Plans 


FAA-STD-036 


Preparation  of  Project  Implementation 
Plans 


I 


EEE  STD  129 


Federal  Information  Resources  Management 
Regulation 


A  Glossary  of  Software  Engineering 
Termlnolo 


U. S. Government  Printing  Office  Style  Guide 


MIL-H-46855 


Human  Engineering  Requirements  for 
Military  Systems,  Equipment  and  Facilities 


MIL-STD-1412C 


Human  Engineering  Design  Criteria  for 
Military  Systems 


MIL-STP-1521B 


Technical  Reviews  and  Audits  for  Systems, 
Equipments,  and  Computer  Software 


MIL-STD- 1  81  5A 


Ada  Programming  Language 


MIL-STO-2Q16 


Automated  Test  Equipment 


MIL  -STD-2011 


Test  Program  Set  Development 


MIL-STD-216S 


Testability  Program  for  Electronic  Systems 
and  Equipment 


M1L-STD-481A 


Configuration  Control  -  Engineering 
Changes,  Deviations  t  Waivers 


MIL-STD-482 


Configuration  Status  Accountin 


MIL-STD-483 


Configuration  Management  Practices  for 
Systems,  Equipment,  Munitions,  and 
Computer  Programs;  21  March  1919 


MIL-STD-490A 


Specification  Practices;  4  June  1985 


MIL-STD-499A 


Engineering  Management 


NAS-MD-UO 


Terms  and  Definitions  for  the  NAS 


Federal  Procurement  Regulations  11.301.1 
through  11.301.5 


WA  0000. 4H 


Washington  Headquarters  Directives 
Checklist  as  of  February  1,  1989 
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